[855] Add the 0.4.0 release notes and a Spark runtime quickstart with a runnable demo by vinishjail97 · Pull Request #913 · apache/incubator-xtable

vinishjail97 · 2026-08-25T00:33:10Z

What

Website documentation plus one runnable demo. No module sources are touched.

0.4.0-incubating release page (website/releases/release-0.4.0-incubating.mdx) and a matching entry on the Downloads page.
New docs page Run a sync on Apache Spark (website/docs/spark-runtime.md), under Quick Start in the sidebar and linked from Installation.
New demo demo/spark-runtime/ — a Spark job that keeps a table readable in the other formats from inside the job that writes it.

This replaces #847, which was closed so the docs could be rewritten and get a fresh review.

The Spark quickstart

xtable-spark-runtime_2.12 0.4.0-incubating is on Maven Central, so the page points at the released jar rather than a source build. The module is not on main, so nothing here depends on it being in the tree.

The page leads with the case the jar exists for. A job already writes a table, adds the jar with --jars, and calls XTableSyncService after its write:

$SPARK_HOME/bin/spark-submit \
  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
  --class com.example.OrdersJob \
  orders-job.jar

The standalone spark-submit entry point and the multi-table --datasetconfig follow, then the CLI options, the format matrix and the Spark version table.

On `--jars` vs `--packages`

Every dependency in the published pom is provided or test scoped, so --packages on this jar resolves one jar and no transitives. Both flags work. --jars is the recommendation only because it needs no Maven Central lookup on each submit.

The real hazard is adding the engines with --packages. --packages and --jars both load into a child class loader, and a second Avro or Parquet there breaks casts across the boundary. Engine jars belong on spark.driver.extraClassPath, which is what ITXTableSparkRuntimeBundle already does. The page and the demo README both say this.

The demo

demo/spark-runtime/XTableInJobSyncDemo.scala runs both directions and checks the row counts, so it doubles as a smoke test of a release:

Direction	The job writes	XTable adds
1	`HUDI`	`ICEBERG`, `DELTA`
2	`ICEBERG`	`HUDI`, `DELTA`

It needs no Docker and no Scala compiler — spark-shell -i compiles the script, so a reader runs one command. The jar comes from Maven Central with curl and goes on --jars.

A local Spark distribution ships no engines, so the README resolves them from the downloaded pom with dependency:build-classpath and puts them on extraClassPath. Spark 3.4.x is required, because Hudi 0.14.0 publishes no Spark 3.5 bundle for the datasource direction 1 writes with; the runtime jar itself supports 3.4 and 3.5.

Verification

Ran the demo README verbatim from a clean directory against Spark 3.4.2 with the published 0.4.0-incubating jar:

[PASS] hudi -> iceberg        1000 rows
[PASS] hudi -> delta          1000 rows
[PASS] iceberg -> hudi        1000 rows
[PASS] iceberg -> delta       1000 rows
ALL CHECKS PASSED

npm run build in website/ succeeds and both new pages generate.
apache-rat:check passes on the new files.

Adds website/releases/release-0.4.0-incubating.mdx and links it from the Downloads page, alongside the source release, its signature and checksum on dlcdn. The page leads with xtable-spark-runtime, the one new artifact the release publishes to Maven Central, and groups the rest into new source formats (Paimon, Parquet), the Delta Kernel target, conversion fixes, and build work. It also lists the three open license documentation issues (apache#700, apache#701, apache#865) that were footnoted during the release vote.

xtable-spark-runtime_2.12 0.4.0-incubating is on Maven Central, so the guide points at the released jar instead of a source build. The module itself is not on main, so nothing here depends on it being in the tree. The page leads with the case the jar exists for: a Spark job that already writes a table adds the jar with --jars and calls XTableSyncService after its write. The standalone spark-submit entry point and the multi-table --datasetconfig follow, then the CLI options, the format matrix and the Spark version table. It also explains why --jars is the recommendation. Every dependency in the published pom is provided or test scoped, so --packages resolves the one jar and no transitives; --jars simply avoids a Maven Central lookup on each submit. The real hazard is adding the engines with --packages, which loads a second Avro or Parquet into a child class loader and breaks casts across the boundary, so engine jars belong on extraClassPath. Linked from the docs sidebar under Quick Start and from the Installation page.

demo/spark-runtime runs the case the guide describes, and checks it. A Spark job writes a table, calls XTableSyncService from inside the same job, and then reads the table back in the other formats and compares row counts. Both directions run: Hudi to Iceberg/Delta, and Iceberg to Hudi/Delta. The demo needs no Docker and no Scala compiler. spark-shell -i compiles the script, so a reader runs one command. The jar comes from Maven Central with curl and goes on --jars, exactly as it would on a job they already run. A local Spark distribution ships no engines, so the README resolves them from the downloaded pom with dependency:build-classpath and puts them on extraClassPath, matching how a cluster supplies them and how ITXTableSparkRuntimeBundle runs. Spark 3.4.x is required because Hudi 0.14.0 publishes no Spark 3.5 bundle for the datasource that direction 1 writes with; the runtime jar itself supports 3.4 and 3.5. Verified end to end against Spark 3.4.2 with the published 0.4.0-incubating jar. All four checks pass.

Renames the guide to how-to-spark-runtime so it matches how-to and how-to-catalog-sync, and retitles it "Run an XTable sync on Apache Spark". Drops two sections that documented Spark itself rather than XTable: the --jars vs --packages comparison, and "Read a Hudi target". The Hudi metadata read option is already covered on the Apache Spark page, which Next steps now links to. Replaces the invented dataset YAML with the config from the project README, and says outright that it is the same file RunSync takes. Adds a short note that the runtime jar addresses tables by path and has no equivalent of the RunSync Iceberg catalog config, which is why an Iceberg source needs dataPath. Release notes: the Delta Kernel entry now covers the source as well as the target, since DeltaKernelConversionSource landed in apache#729; the checkpoint CRC bullet says fail fast; the Hudi additional-properties bullet is gone. Neither page mentions delta-core any more, and the Spark table names Delta Standalone and Delta Kernel instead. Prose across all files follows the Hudi and Iceberg docs voice rather than clipped one-clause sentences, and consistently says a table is interoperable rather than readable or queryable. The demo banners changed with that wording, so the demo was re-run end to end against Spark 3.4.2 with the published 0.4.0-incubating jar. All four checks still pass, and the expected output in the README is from that run.

<basePath>/data is what the Hadoop catalog happens to produce, not something Iceberg guarantees; write.data.path and object-storage layouts can place data files anywhere. Both the guide and the demo comment stated it as a rule. The guide now frames dataPath by what it is for: set it when a table's data files do not sit directly under basePath, because that is where each target writes its metadata. The demo comment scopes its <basePath>/data claim to the catalog the demo configures. No behaviour change; dataPath was always a free-form path.

kroushan-nit

LGTM

rangareddy · 2026-08-25T04:28:08Z

+Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job


Move 'Adding the sync to a Spark job' after 'Running a sync as its own job'.

Reason: The current order causes confusion for new users because references like orders-job.jar are introduced without context.

rangareddy · 2026-08-25T04:29:26Z

+This is what the jar is for. Your job already writes a table in one format, and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application jar:


Please update the structure so it first covers building the orders-job, and then details how to run it using spark-submit.

rangareddy · 2026-08-25T04:30:44Z

+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";


In the other quick-start examples, we used local paths, but here an S3 path is used instead. Could we update this to keep it consistent?

rangareddy · 2026-08-25T04:32:21Z

+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:


Could we mention that users need to create the Hudi table using the Quickstart guide before running spark-submit?

rangareddy · 2026-08-25T04:35:42Z

+The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:
+
+```shell md title="shell"


Can we add a section on running XTableSparkSync with and without a configuration file or single-table vs. multi-table syncs?

rangareddy · 2026-08-25T04:41:21Z

+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is


when i click on link, '404 - page not found' error is dispalyed.

rangareddy · 2026-08-25T04:42:48Z


 To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS).

+### Release 0.4.0-incubating


We need to add the links for each release hyper link.

rangareddy · 2026-08-25T04:42:55Z

+* Release Notes: [Release Notes 0.4.0-incubating](release-0.4.0-incubating.mdx)
+* Maven Search: [Apache XTable 0.4.0-incubating](https://central.sonatype.com/search?q=g:org.apache.xtable%20v:0.4.0-incubating&smo=true)
+
 ### Release 0.3.0-incubating


We need to add the links for each release hyper link.

rangareddy · 2026-08-25T04:45:03Z

Can we move 'Downloads' to the top, right after 'Latest Releases'?

rangareddy · 2026-08-25T04:48:13Z

+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync.
+
+### 🆕 Delta Kernel


Could we include a callout/note here to explicitly state Spark 3.5 support?

vinishjail97 added 3 commits August 24, 2026 17:32

vinishjail97 commented Aug 25, 2026

View reviewed changes

Comment thread website/docs/how-to-spark-runtime.md Outdated

vinishjail97 requested a review from the-other-tim-brown August 25, 2026 01:22

kroushan-nit approved these changes Aug 25, 2026

View reviewed changes

slachiewicz approved these changes Aug 25, 2026

View reviewed changes

the-other-tim-brown approved these changes Aug 25, 2026

View reviewed changes

vinishjail97 merged commit c205598 into apache:main Aug 25, 2026
6 checks passed

vinishjail97 mentioned this pull request Aug 25, 2026

[855] Add the 0.4.0 release blog post and widen the blog layout #915

Merged

rangareddy reviewed Aug 26, 2026

View reviewed changes


		To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS).

		### Release 0.4.0-incubating

Uh oh!

Conversation

vinishjail97 commented Aug 25, 2026

What

The Spark quickstart

On --jars vs --packages

The demo

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kroushan-nit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

On `--jars` vs `--packages`