End to End xtable to Trino query with source iceberg and target hudi and Delta Lake with HMS and S3 #463

alberttwong · 2024-06-07T18:39:19Z

alberttwong
Jun 7, 2024

Using openjdk 11, spark 3.4, hive 2.3.10, hadoop 2.10.2

using https://github.com/alberttwong/incubator-xtable/tree/main/demo-s3 as the base

alberttwong · 2024-06-07T18:39:25Z

alberttwong
Jun 7, 2024
Author

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-client:2.10.2,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2 \
    --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
    --conf spark.sql.defaultCatalog=iceberg \
    --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.iceberg.warehouse=s3://warehouse \
    --conf spark.sql.catalog.iceberg.type=hive

0 replies

alberttwong · 2024-06-07T19:01:45Z

alberttwong
Jun 7, 2024
Author

spark-sql ()> CREATE TABLE iceberg_db.taxis 
            > (
            >   vendor_id bigint,
            >   trip_id bigint,
            >   trip_distance float,
            >   fare_amount double,
            >   store_and_fwd_flag string
            > )
            > PARTITIONED BY (vendor_id) ;
24/06/07 19:01:16 ERROR SparkSQLDriver: Failed in [CREATE TABLE iceberg_db.taxis 
(
  vendor_id bigint,
  trip_id bigint,
  trip_distance float,
  fare_amount double,
  store_and_fwd_flag string
)
PARTITIONED BY (vendor_id) ]
software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Code: 301, Request ID: E0FK57YQCA0RDX2A, Extended Request ID: bS92IKcb8KbhjrzTBBee08q2kkiNg2cYp+jOX6RldUZ8BrEhFwDz0GoU7LzRwnMRZ1HBvUinByM=)

1 reply

alberttwong Jun 7, 2024
Author

don't use --conf spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO. used a different spark-sql command

alberttwong · 2024-06-07T19:49:45Z

alberttwong
Jun 7, 2024
Author

CREATE SCHEMA iceberg_db LOCATION 's3a://warehouse/';
CREATE TABLE iceberg_db.taxis 
(
  vendor_id bigint,
  trip_id bigint,
  trip_distance float,
  fare_amount double,
  store_and_fwd_flag string
)
PARTITIONED BY (vendor_id) ;
INSERT INTO iceberg_db.taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');

0 replies

alberttwong · 2024-06-07T19:56:43Z

alberttwong
Jun 7, 2024
Author

Trino shows the data.

sh-5.1$ trino
trino> select * from iceberg.iceberg_db.taxis;
 vendor_id | trip_id | trip_distance | fare_amount | store_and_fwd_flag 
-----------+---------+---------------+-------------+--------------------
         1 | 1000371 |           1.8 |       15.32 | N                  
         1 | 1000374 |           8.4 |       42.13 | Y                  
         2 | 1000372 |           2.5 |       22.15 | N                  
         2 | 1000373 |           0.9 |        9.01 | N                  
(4 rows)

Query 20240607_200019_00001_ynf4i, FINISHED, 1 node
Splits: 2 total, 2 done (100.00%)
0.29 [4 rows, 3KB] [13 rows/s, 10.4KB/s]

spark-sql shows the data

spark-sql ()> use iceberg_db;
Time taken: 0.091 seconds
spark-sql (iceberg_db)> show tables;
taxis
Time taken: 0.253 seconds, Fetched 1 row(s)
spark-sql (iceberg_db)> select * from taxis;
1       1000371 1.8     15.32   N
1       1000374 8.4     42.13   Y
2       1000372 2.5     22.15   N
2       1000373 0.9     9.01    N

files in s3 show correctly

sh-5.1# mc alias set warehouse http://minio:9000 admin password
Added `warehouse` successfully.
sh-5.1# mc ls -r warehouse
[2024-06-07 20:01:59 UTC]     0B STANDARD warehouse/taxis/_delta_log/
[2024-06-07 20:00:59 UTC] 1.5KiB STANDARD warehouse/taxis/data/vendor_id=1/00000-10-e0fc3ef1-3606-4591-bcf6-d72b25747380-0-00001.parquet
[2024-06-07 19:59:51 UTC] 1.5KiB STANDARD warehouse/taxis/data/vendor_id=1/00000-5-865f6992-e612-49a0-a8db-a27fd0f7d02a-0-00001.parquet
[2024-06-07 20:00:59 UTC] 1.5KiB STANDARD warehouse/taxis/data/vendor_id=2/00000-10-e0fc3ef1-3606-4591-bcf6-d72b25747380-0-00002.parquet
[2024-06-07 19:59:51 UTC] 1.5KiB STANDARD warehouse/taxis/data/vendor_id=2/00000-5-865f6992-e612-49a0-a8db-a27fd0f7d02a-0-00002.parquet
[2024-06-07 19:59:24 UTC] 1.4KiB STANDARD warehouse/taxis/metadata/00000-77bdf818-507f-48f1-971a-1898c294bf49.metadata.json
[2024-06-07 19:59:51 UTC] 2.4KiB STANDARD warehouse/taxis/metadata/00001-ef7e1726-2ec0-4582-bef8-9c37f96e2909.metadata.json
[2024-06-07 20:00:59 UTC] 3.4KiB STANDARD warehouse/taxis/metadata/00002-15d50d43-f5d8-4faa-b2b7-f41aca3f758f.metadata.json
[2024-06-07 19:59:51 UTC] 7.0KiB STANDARD warehouse/taxis/metadata/ae7384d8-6b4a-4fd6-bbc2-e9b621ba9e0b-m0.avro
[2024-06-07 20:00:59 UTC] 7.0KiB STANDARD warehouse/taxis/metadata/ebef7e50-73aa-4428-98aa-6ad0a8ed7802-m0.avro
[2024-06-07 19:59:51 UTC] 4.1KiB STANDARD warehouse/taxis/metadata/snap-1202211864160811787-1-ae7384d8-6b4a-4fd6-bbc2-e9b621ba9e0b.avro
[2024-06-07 20:00:59 UTC] 4.2KiB STANDARD warehouse/taxis/metadata/snap-3391991307049980362-1-ebef7e50-73aa-4428-98aa-6ad0a8ed7802.avro

root@spark:/opt/xtable/jars# export AWS_SECRET_ACCESS_KEY=password
root@spark:/opt/xtable/jars# export AWS_ACCESS_KEY_ID=admin
root@spark:/opt/xtable/jars# export ENDPOINT=http://minio:9000
root@spark:/opt/xtable/jars# export AWS_REGION=us-east-1
root@spark:/opt/xtable/jars# cd /opt/xtable/jars/; java -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig xtable_iceberg.yaml -p core-site.xml
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
2024-06-07 19:54:51 INFO  org.apache.xtable.utilities.RunSync:148 - Running sync for basePath s3a://warehouse/taxis for following table formats [HUDI, DELTA]
2024-06-07 19:54:51 INFO  org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/taxis
2024-06-07 19:54:51 WARN  org.apache.hadoop.util.NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-07 19:54:51 WARN  org.apache.hadoop.metrics2.impl.MetricsConfig:136 - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-06-07 19:54:52 WARN  org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2024-06-07 19:54:52 INFO  org.apache.xtable.hudi.HudiTableManager:73 - Hudi table does not exist, will be created on first sync
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/xtable/jars/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-07 19:54:53 INFO  org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore `LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore)` is used for scheme `s3a`
2024-06-07 19:54:53 INFO  org.apache.spark.sql.delta.DeltaLog:60 - Creating initial snapshot without metadata, because the directory is empty
2024-06-07 19:54:54 INFO  org.apache.spark.sql.delta.InitialSnapshot:60 - [tableId=95c3e958-7fec-4917-bb9d-28bdf4504d33] Created snapshot InitialSnapshot(path=s3a://warehouse/taxis/_delta_log, version=-1, metadata=Metadata(bd69b4e8-e7de-4df2-b8fe-6ada0e1d0cc8,null,null,Format(parquet,Map()),null,List(),Map(),Some(1717790094004)), logSegment=LogSegment(s3a://warehouse/taxis/_delta_log,-1,List(),None,-1), checksumOpt=None)
2024-06-07 19:54:54 INFO  org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync.
2024-06-07 19:54:54 INFO  org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync.
2024-06-07 19:54:54 WARN  org.apache.iceberg.hadoop.HadoopTableOperations:325 - Error reading version hint file s3a://warehouse/taxis/metadata/version-hint.text
java.io.FileNotFoundException: No such file or directory: s3a://warehouse/taxis/metadata/version-hint.text
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3801) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:5288) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$executeOpen$6(S3AFileSystem.java:1578) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.executeOpen(S3AFileSystem.java:1576) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1550) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:997) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.iceberg.hadoop.HadoopTableOperations.findVersion(HadoopTableOperations.java:318) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.iceberg.hadoop.HadoopTableOperations.refresh(HadoopTableOperations.java:104) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.iceberg.hadoop.HadoopTableOperations.current(HadoopTableOperations.java:84) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:94) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergTableManager.lambda$getTable$1(IcebergTableManager.java:58) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at java.util.Optional.orElseGet(Unknown Source) [?:?]
        at org.apache.xtable.iceberg.IcebergTableManager.getTable(IcebergTableManager.java:58) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.initSourceTable(IcebergConversionSource.java:81) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.getSourceTable(IcebergConversionSource.java:60) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:121) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:183) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:121) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.utilities.RunSync.main(RunSync.java:169) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
2024-06-07 19:54:54 ERROR org.apache.xtable.utilities.RunSync:171 - Error running sync for s3a://warehouse/taxis
org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at location: s3a://warehouse/taxis
        at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:97) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergTableManager.lambda$getTable$1(IcebergTableManager.java:58) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at java.util.Optional.orElseGet(Unknown Source) ~[?:?]
        at org.apache.xtable.iceberg.IcebergTableManager.getTable(IcebergTableManager.java:58) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.initSourceTable(IcebergConversionSource.java:81) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.getSourceTable(IcebergConversionSource.java:60) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:183) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
        at org.apache.xtable.utilities.RunSync.main(RunSync.java:169) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]

1 reply

alberttwong Jun 7, 2024
Author

hint? #431 (comment). switching to hadoop type

alberttwong · 2024-06-07T20:51:39Z

alberttwong
Jun 7, 2024
Author

    spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-client:2.10.2,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2 \
    --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
    --conf spark.sql.defaultCatalog=iceberg \
    --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.iceberg.warehouse=s3://warehouse \
    --conf spark.sql.catalog.iceberg.type=hadoop

error

Exception in thread "main" org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file system for path: s3://warehouse
        at org.apache.iceberg.hadoop.Util.getFs(Util.java:58)
        at org.apache.iceberg.hadoop.HadoopCatalog.initialize(HadoopCatalog.java:112)
        at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:255)
        at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:309)
        at org.apache.iceberg.spark.SparkCatalog.buildIcebergCatalog(SparkCatalog.java:154)
        at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:751)
        at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:65)
        at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:53)
        at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
        at org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:53)
        at org.apache.spark.sql.connector.catalog.CatalogManager.currentCatalog(CatalogManager.scala:122)
        at org.apache.spark.sql.connector.catalog.CatalogManager.currentNamespace(CatalogManager.scala:93)
        at org.apache.spark.sql.internal.CatalogImpl.currentDatabase(CatalogImpl.scala:65)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.currentDB$1(SparkSQLCLIDriver.scala:285)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.promptWithCurrentDB$1(SparkSQLCLIDriver.scala:292)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:296)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
        at org.apache.iceberg.hadoop.Util.getFs(Util.java:56)

1 reply

alberttwong Jun 7, 2024
Author

have to use s3a

    spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-client:2.10.2,com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2 \
    --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
    --conf spark.sql.defaultCatalog=iceberg \
    --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.iceberg.warehouse=s3a://warehouse \
    --conf spark.sql.catalog.iceberg.type=hadoop

alberttwong · 2024-06-07T21:06:51Z

alberttwong
Jun 7, 2024
Author

spark-sql ()> show schemas;
24/06/07 21:06:27 ERROR SparkSQLDriver: Failed in [show schemas]
java.lang.IllegalArgumentException: path must be absolute
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
        at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:147)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1876)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1812)
        at org.apache.iceberg.hadoop.HadoopCatalog.isDirectory(HadoopCatalog.java:176)
        at org.apache.iceberg.hadoop.HadoopCatalog.isNamespace(HadoopCatalog.java:377)
        at org.apache.iceberg.hadoop.HadoopCatalog.listNamespaces(HadoopCatalog.java:307)
        at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74)
        at org.apache.iceberg.spark.SparkCatalog.listNamespaces(SparkCatalog.java:434)
        at org.apache.spark.sql.execution.datasources.v2.ShowNamespacesExec.run(ShowNamespacesExec.scala:42)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
        at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:218)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:95)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:640)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:630)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:671)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:415)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:533)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:527)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:527)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:307)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
path must be absolute
java.lang.IllegalArgumentException: path must be absolute
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60)
        at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56)
        at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:147)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1876)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1812)
        at org.apache.iceberg.hadoop.HadoopCatalog.isDirectory(HadoopCatalog.java:176)
        at org.apache.iceberg.hadoop.HadoopCatalog.isNamespace(HadoopCatalog.java:377)
        at org.apache.iceberg.hadoop.HadoopCatalog.listNamespaces(HadoopCatalog.java:307)
        at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74)
        at org.apache.iceberg.spark.SparkCatalog.listNamespaces(SparkCatalog.java:434)
        at org.apache.spark.sql.execution.datasources.v2.ShowNamespacesExec.run(ShowNamespacesExec.scala:42)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
        at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
        at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:218)
        at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:95)
        at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:640)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:630)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:671)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:415)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:533)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:527)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:527)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:307)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2 replies

alberttwong Jun 7, 2024
Author

I couldn't figure it out but used a different spark-sql

spark-sql --packages com.amazonaws:aws-java-sdk-s3:1.11.271,org.apache.hadoop:hadoop-aws:2.10.2,org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2 \
--conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \
--conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog" \
--conf "spark.sql.catalog.spark_catalog.type=hadoop" \
--conf "spark.sql.catalog.spark_catalog.warehouse=s3a://warehouse" \
--conf "spark.sql.catalog.hive_prod=org.apache.iceberg.spark.SparkCatalog" \
--conf "spark.sql.catalog.hive_prod.type=hadoop" \
--conf "spark.sql.catalog.hive_prod.warehouse=s3a://warehouse" \
--conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"

alberttwong Jun 8, 2024
Author

this lets me go but I can't find a type=hadoop that works.

alberttwong · 2024-06-07T23:36:39Z

alberttwong
Jun 7, 2024
Author

CREATE SCHEMA iceberg_db LOCATION 's3a://warehouse/';
CREATE TABLE taxis 
(
  vendor_id bigint,
  trip_id bigint,
  trip_distance float,
  fare_amount double,
  store_and_fwd_flag string
)
USING iceberg
PARTITIONED BY (vendor_id) ;
INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');

2 replies

alberttwong Jun 8, 2024
Author

24/06/08 03:56:22 ERROR SparkSQLDriver: Failed in [show schemas]
java.lang.IllegalArgumentException: path must be absolute

alberttwong Jun 8, 2024
Author

this seems like a dead end. I can't find a type=hadoop that works.

alberttwong · 2024-06-26T16:59:37Z

alberttwong
Jun 26, 2024
Author

summary of issue... xtable needs iceberg to generate metadata/version-hint.text to convert to hudi and delta.

0 replies

alberttwong · 2024-07-16T00:25:27Z

alberttwong
Jul 16, 2024
Author

cannot figure how to use hadoop 2.x libraries with spark 3.4. Works fine if you use hadoop 3.x libraries with spark 3.4

export AWS_SECRET_ACCESS_KEY=password
export AWS_ACCESS_KEY_ID=admin
export ENDPOINT=http://minio:9000
export AWS_REGION=us-east-1
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-aws:3.3.4,software.amazon.awssdk:bundle:2.17.178,software.amazon.awssdk:url-connection-client:2.17.178 \
    --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
    --conf spark.sql.defaultCatalog=iceberg \
    --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.iceberg.warehouse=s3a://warehouse \
    --conf spark.sql.catalog.iceberg.type=hadoop

1 reply

alberttwong Jul 16, 2024
Author

don't use the create schema sql command.

CREATE TABLE taxis 
(
  vendor_id bigint,
  trip_id bigint,
  trip_distance float,
  fare_amount double,
  store_and_fwd_flag string
)
USING iceberg
PARTITIONED BY (vendor_id) ;
INSERT INTO taxis VALUES (1, 1000371, 1.8, 15.32, 'N'), (2, 1000372, 2.5, 22.15, 'N'), (2, 1000373, 0.9, 9.01, 'N'), (1, 1000374, 8.4, 42.13, 'Y');

alberttwong · 2024-07-16T17:09:17Z

alberttwong
Jul 16, 2024
Author

export AWS_SECRET_ACCESS_KEY=password
export AWS_ACCESS_KEY_ID=admin
export ENDPOINT=http://minio:9000
export AWS_REGION=us-east-1
cd /opt/xtable/jars/; java -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig xtable_iceberg.yaml -p core-site.xml

1 reply

alberttwong Jul 16, 2024
Author

root@spark:/opt/xtable/jars# cat xtable_iceberg.yaml 
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
sourceFormat: ICEBERG
targetFormats:
  - HUDI
  - DELTA
datasets:
  -
    tableBasePath: s3a://warehouse/taxis
    tableName: taxis
    partitionSpec: vendor_id:VALUE

Uh oh!

End to End xtable to Trino query with source iceberg and target hudi and Delta Lake with HMS and S3 #463

Uh oh!

Uh oh!

alberttwong Jun 7, 2024

Replies: 10 comments · 9 replies

Uh oh!

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 8, 2024 Author

Uh oh!

alberttwong Jun 7, 2024 Author

Uh oh!

alberttwong Jun 8, 2024 Author

Uh oh!

alberttwong Jun 8, 2024 Author

Uh oh!

alberttwong Jun 26, 2024 Author

Uh oh!

Uh oh!

alberttwong Jul 16, 2024 Author

Uh oh!

alberttwong Jul 16, 2024 Author

Uh oh!

alberttwong Jul 16, 2024 Author

Uh oh!

alberttwong Jul 16, 2024 Author

alberttwong
Jun 7, 2024

Replies: 10 comments 9 replies

alberttwong
Jun 7, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong Jun 7, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong Jun 7, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong Jun 7, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong Jun 7, 2024
Author

alberttwong Jun 8, 2024
Author

alberttwong
Jun 7, 2024
Author

alberttwong Jun 8, 2024
Author

alberttwong Jun 8, 2024
Author

alberttwong
Jun 26, 2024
Author

alberttwong
Jul 16, 2024
Author

alberttwong Jul 16, 2024
Author

alberttwong
Jul 16, 2024
Author

alberttwong Jul 16, 2024
Author