Skip to content

Commit 8248b3f

Browse files
vinishjail97vinishjail97
committed
[docs] Address review feedback on the 0.4.0 docs
Renames the guide to how-to-spark-runtime so it matches how-to and how-to-catalog-sync, and retitles it "Run an XTable sync on Apache Spark". Drops two sections that documented Spark itself rather than XTable: the --jars vs --packages comparison, and "Read a Hudi target". The Hudi metadata read option is already covered on the Apache Spark page, which Next steps now links to. Replaces the invented dataset YAML with the config from the project README, and says outright that it is the same file RunSync takes. Adds a short note that the runtime jar addresses tables by path and has no equivalent of the RunSync Iceberg catalog config, which is why an Iceberg source needs dataPath. Release notes: the Delta Kernel entry now covers the source as well as the target, since DeltaKernelConversionSource landed in #729; the checkpoint CRC bullet says fail fast; the Hudi additional-properties bullet is gone. Neither page mentions delta-core any more, and the Spark table names Delta Standalone and Delta Kernel instead. Prose across all files follows the Hudi and Iceberg docs voice rather than clipped one-clause sentences, and consistently says a table is interoperable rather than readable or queryable. The demo banners changed with that wording, so the demo was re-run end to end against Spark 3.4.2 with the published 0.4.0-incubating jar. All four checks still pass, and the expected output in the README is from that run.
1 parent 1fd45a8 commit 8248b3f

8 files changed

Lines changed: 260 additions & 283 deletions

File tree

‎demo/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ When you are done, use `./stop_demo.sh` to stop and remove the containers. Pass
2727
- `work/demo.ipynb` — the main demo: two existing datasets (Hudi and Delta) are synced across Hudi, Delta and Iceberg, updated from Trino, joined, and validated from Trino.
2828

2929
## Standalone Spark demo
30-
- `spark-runtime/` — a Spark job that keeps a table readable in the other formats from inside the job that writes it, using the `xtable-spark-runtime` jar on `--jars`. It runs Hudi to Iceberg/Delta and Iceberg to Hudi/Delta, and checks the row counts. It needs no Docker; see `spark-runtime/README.md`.
30+
- `spark-runtime/` — keeps a table interoperable by adding the `xtable-spark-runtime` jar with `--jars`. No Docker required; see `spark-runtime/README.md`.
3131

3232
## Accessing Services
3333
### Jupyter Notebook

‎demo/spark-runtime/README.md‎

Lines changed: 15 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -18,26 +18,26 @@
1818

1919
# In-job sync demo for `xtable-spark-runtime`
2020

21-
This demo shows the case the `xtable-spark-runtime` jar exists for. A Spark job already writes a
22-
table. The job adds the jar with `--jars` and calls `XTableSyncService` after the write. The table
23-
then reads in the other formats, and no data is rewritten.
21+
This demo shows what the `xtable-spark-runtime` jar is for: a Spark job that already writes a table
22+
adds the jar with `--jars` and calls `XTableSyncService` right after its write, making the table
23+
interoperable without rewriting any data.
2424

25-
The demo runs both directions and checks the row counts, so it also works as a smoke test of a
26-
release.
25+
It runs the sync in both directions and verifies the row counts, so it also serves as a smoke test
26+
for a release.
2727

2828
| Direction | The job writes | XTable adds |
2929
| --- | --- | --- |
3030
| 1 | `HUDI` | `ICEBERG`, `DELTA` |
3131
| 2 | `ICEBERG` | `HUDI`, `DELTA` |
3232

33-
The demo needs no Docker and no Scala compiler. `spark-shell` compiles the script. This is separate
34-
from the notebook demo in the parent directory.
33+
No Docker and no Scala compiler are needed — `spark-shell` compiles the script for you. This is
34+
separate from the notebook demo in the parent directory.
3535

3636
## Prerequisites
3737

3838
- A Spark **3.4.x** distribution. Hudi 0.14.0 publishes no Spark 3.5 bundle, so the Hudi Spark
3939
datasource that direction 1 writes with needs a 3.4 runtime. The runtime jar itself supports both
40-
Spark 3.4 and Spark 3.5.
40+
Spark 3.4 and 3.5.
4141
- JDK 11.
4242
- Maven, to resolve the engine jars in step 2.
4343

@@ -121,15 +121,15 @@ runtime, the Hudi write in direction 1 fails with
121121

122122
```
123123
==============================================================================
124-
DIRECTION 1: the job writes HUDI -> readable as ICEBERG and DELTA
124+
DIRECTION 1: the job writes HUDI -> interoperable with ICEBERG and DELTA
125125
==============================================================================
126126
wrote the Hudi table at /tmp/xtable-spark-demo/out/hudi_orders
127127
XTableSyncService returned [ICEBERG, DELTA]
128128
[PASS] hudi -> iceberg 1000 rows
129129
[PASS] hudi -> delta 1000 rows
130130
131131
==============================================================================
132-
DIRECTION 2: the job writes ICEBERG -> readable as HUDI and DELTA
132+
DIRECTION 2: the job writes ICEBERG -> interoperable with HUDI and DELTA
133133
==============================================================================
134134
wrote the Iceberg table at /tmp/xtable-spark-demo/out/iceberg_warehouse/db/orders
135135
XTableSyncService returned [HUDI, DELTA]
@@ -146,17 +146,15 @@ Set `XT_DEMO_ROWS` to change the row count. The default is 1000.
146146

147147
## Notes
148148

149-
- **Read a Hudi target with `hoodie.metadata.enable=true`.** XTable records the file listing of a
150-
Hudi target in the Hudi metadata table. Without the option, the sync reports success and the read
151-
returns zero rows.
152-
- **Do not add the engines with `--packages`.** `--packages` and `--jars` load classes in a child
153-
class loader. A second copy of Avro or Parquet there breaks casts across the class loader
154-
boundary. Use `extraClassPath` for engine jars, as this demo does.
149+
- **A Hudi target needs `hoodie.metadata.enable=true` on read.** XTable records a Hudi target's
150+
file listing in the Hudi metadata table, so without this option the sync reports success and the
151+
read comes back empty. See [Apache Spark](https://xtable.apache.org/docs/spark) for the read
152+
options each format needs.
155153
- For paths on S3, add S3A and credentials:
156154

157155
```shell
158156
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
159157
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EnvironmentVariableCredentialsProvider
160158
```
161159

162-
See [Run a sync on Apache Spark](https://xtable.apache.org/docs/spark-runtime) for the full guide.
160+
See [Run an XTable sync on Apache Spark](https://xtable.apache.org/docs/how-to-spark-runtime) for the full guide.

‎demo/spark-runtime/XTableInJobSyncDemo.scala‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
*/
1818

1919
/*
20-
* Keeps a table readable in the other formats from inside the Spark job that writes it.
20+
* Keeps a table interoperable from inside the Spark job that writes it.
2121
* The job adds the xtable-spark-runtime jar with --jars and calls XTableSyncService after
2222
* the write. No data is rewritten.
2323
*
@@ -26,7 +26,7 @@
2626
* 2. the job writes ICEBERG -> XTable adds HUDI and DELTA
2727
*
2828
* See demo/spark-runtime/README.md for how to run it, and
29-
* https://xtable.apache.org/docs/spark-runtime for the guide.
29+
* https://xtable.apache.org/docs/how-to-spark-runtime for the guide.
3030
*
3131
* Environment:
3232
* XT_DEMO_DIR output directory (default /tmp/xtable-demo)
@@ -94,7 +94,7 @@ val orders: DataFrame = spark
9494
.withColumn("ts", lit(System.currentTimeMillis()))
9595
println(s" $rows rows, ${orders.schema.fields.length} columns")
9696

97-
banner("DIRECTION 1: the job writes HUDI -> readable as ICEBERG and DELTA")
97+
banner("DIRECTION 1: the job writes HUDI -> interoperable with ICEBERG and DELTA")
9898
val hudiPath = s"$outPrefix/hudi_orders"
9999

100100
// ---- the write the job already does ------------------------------------------------------
@@ -125,7 +125,7 @@ println(s" XTableSyncService returned ${hudiResults.keySet()}")
125125
check("hudi -> iceberg", rows, "iceberg", hudiPath)
126126
check("hudi -> delta", rows, "delta", hudiPath)
127127

128-
banner("DIRECTION 2: the job writes ICEBERG -> readable as HUDI and DELTA")
128+
banner("DIRECTION 2: the job writes ICEBERG -> interoperable with HUDI and DELTA")
129129
val icebergBase = s"$outPrefix/iceberg_warehouse/db/orders"
130130

131131
// ---- the write the job already does ------------------------------------------------------

‎website/docs/how-to-spark-runtime.md‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
---
2+
sidebar_position: 3
3+
title: "Run an XTable sync on Apache Spark"
4+
---
5+
6+
import Tabs from '@theme/Tabs';
7+
import TabItem from '@theme/TabItem';
8+
9+
# Run an XTable sync on Apache Spark
10+
11+
`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache
12+
Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source
13+
table's metadata and writes the target format's metadata alongside the data that is already there.
14+
15+
The jar is meant to be dropped into a Spark job you already run, so you don't need a separate
16+
process or a separate cluster to keep a table interoperable across formats.
17+
18+
## Getting the jar
19+
20+
The jar is published to Maven Central as
21+
`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once:
22+
23+
```shell md title="shell"
24+
curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
25+
```
26+
27+
Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and
28+
Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the
29+
full list of releases.
30+
31+
## Adding the sync to a Spark job
32+
33+
This is what the jar is for. Your job already writes a table in one format, and you want that same
34+
table to be interoperable with the others as soon as the write finishes.
35+
36+
Add the jar to the `spark-submit` you already use, alongside your application jar:
37+
38+
```shell md title="shell"
39+
$SPARK_HOME/bin/spark-submit \
40+
--jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
41+
--class com.example.OrdersJob \
42+
orders-job.jar
43+
```
44+
45+
Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and
46+
hand it the session's Hadoop configuration:
47+
48+
<Tabs
49+
groupId="language"
50+
defaultValue="scala"
51+
values={[
52+
{ label: 'Scala', value: 'scala', },
53+
{ label: 'Java', value: 'java', },
54+
]}
55+
>
56+
<TabItem value="scala">
57+
58+
```scala md title="OrdersJob.scala"
59+
import java.util.{Arrays => JArrays}
60+
import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
61+
62+
val basePath = "s3://example-warehouse/db/orders"
63+
64+
// the write your job already does
65+
df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
66+
67+
// the one call you add
68+
new XTableSyncService().sync(
69+
TableSyncSpec.builder()
70+
.key("orders")
71+
.basePath(basePath)
72+
.sourceFormat("HUDI")
73+
.targets(JArrays.asList("ICEBERG", "DELTA"))
74+
.build(),
75+
spark.sparkContext.hadoopConfiguration)
76+
```
77+
78+
</TabItem>
79+
<TabItem value="java">
80+
81+
```java md title="OrdersJob.java"
82+
import java.util.Arrays;
83+
import org.apache.xtable.spark.TableSyncSpec;
84+
import org.apache.xtable.spark.XTableSyncService;
85+
86+
String basePath = "s3://example-warehouse/db/orders";
87+
88+
// the write your job already does
89+
df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
90+
91+
// the one call you add
92+
new XTableSyncService()
93+
.sync(
94+
TableSyncSpec.builder()
95+
.key("orders")
96+
.basePath(basePath)
97+
.sourceFormat("HUDI")
98+
.targets(Arrays.asList("ICEBERG", "DELTA"))
99+
.build(),
100+
spark.sparkContext().hadoopConfiguration());
101+
```
102+
103+
</TabItem>
104+
</Tabs>
105+
106+
To compile against these classes, add the same Maven coordinates to your build with `provided`
107+
scope.
108+
109+
The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling
110+
back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That
111+
makes it safe to call after every write.
112+
113+
:::note Tables are addressed by path
114+
The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent
115+
of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the
116+
table root and set `dataPath` to `<basePath>/data`, since that is where Iceberg keeps its data
117+
files and where each target writes its metadata.
118+
:::
119+
120+
:::tip Runnable example
121+
[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is
122+
a complete job that syncs both directions and verifies the row counts.
123+
:::
124+
125+
## Running a sync as its own job
126+
127+
The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this
128+
bundle. Pass the jar as the application jar rather than with `--jars`:
129+
130+
```shell md title="shell"
131+
$SPARK_HOME/bin/spark-submit \
132+
--class org.apache.xtable.spark.XTableSparkSync \
133+
--master 'local[*]' \
134+
xtable-spark-runtime_2.12-0.4.0-incubating.jar \
135+
--basepath /path/to/hudi_table \
136+
--sourceformat HUDI \
137+
--targets ICEBERG,DELTA
138+
```
139+
140+
To sync several tables in one submit, use `--datasetconfig`. It takes the same YAML that `RunSync`
141+
uses, so an existing config works unchanged, and unlike `RunSync` the config itself may live on
142+
cloud storage:
143+
144+
```yaml md title="my_config.yaml"
145+
sourceFormat: HUDI
146+
targetFormats:
147+
- DELTA
148+
- ICEBERG
149+
datasets:
150+
-
151+
tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
152+
tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data
153+
tableName: call_center
154+
namespace: my.db
155+
-
156+
tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
157+
tableName: catalog_sales
158+
partitionSpec: cs_sold_date_sk:VALUE
159+
-
160+
tableBasePath: s3://hudi/multi-partition-dataset
161+
tableName: multi_partition_dataset
162+
partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE
163+
```
164+
165+
```shell md title="shell"
166+
$SPARK_HOME/bin/spark-submit \
167+
--class org.apache.xtable.spark.XTableSparkSync \
168+
xtable-spark-runtime_2.12-0.4.0-incubating.jar \
169+
--datasetconfig my_config.yaml
170+
```
171+
172+
`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the other.
173+
174+
| Option | Description |
175+
| --- | --- |
176+
| `--basepath` | Base path of the source table. |
177+
| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. |
178+
| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. |
179+
| `--datasetconfig` | Path to a YAML config listing several tables. May be local or on cloud storage. |
180+
| `--datapath` | Path to the data files, when it differs from the base path. |
181+
| `--tablename` | Table name. Defaults to the last segment of the base path. |
182+
| `--namespace` | Dot-separated table namespace. |
183+
| `--partitionspec` | Hudi source partition field spec, for example `level:VALUE`. |
184+
| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. |
185+
| `--help` | Print the usage text. |
186+
187+
## Supported formats
188+
189+
Paimon and Parquet are read-only sources; XTable does not write either format as a target.
190+
191+
| Source ↓ / Target → | Hudi | Iceberg | Delta |
192+
| --- | :---: | :---: | :---: |
193+
| **Hudi** | – | ✅ | ✅ |
194+
| **Iceberg** | ✅ | – | ✅ |
195+
| **Delta** | ✅ | ✅ | – |
196+
| **Paimon** | ✅ | ✅ | ✅ |
197+
| **Parquet** | ✅ | ✅ | ✅ |
198+
199+
## Spark version support
200+
201+
Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the
202+
Spark versions below. Delta is the only engine whose implementation depends on the Spark version,
203+
and the jar chooses the right one automatically:
204+
205+
| Spark version | Hudi and Iceberg | Delta implementation |
206+
| --- | :---: | --- |
207+
| 3.4.x | ✅ | Delta Standalone |
208+
| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically |
209+
210+
Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is routed
211+
through Delta Kernel with no flag needed. If you want Kernel on Spark 3.4 as well, pass
212+
`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`.
213+
214+
## Next steps
215+
216+
- See the [Quickstart](/docs/how-to) for an end-to-end interoperability walkthrough.
217+
- See [Apache Spark](/docs/spark) for the options each format needs when you query a synced table.
218+
- See [Installation](/docs/setup) if you'd rather build the project from source.

‎website/docs/setup.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,5 +25,5 @@ For more information on the steps, follow the project's GitHub [README.md](https
2525
See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between
2626
different table formats.
2727

28-
To run a sync on an Apache Spark cluster you already have, see
29-
[Run a sync on Apache Spark](/docs/spark-runtime).
28+
To run a sync on a Spark cluster you already have, see
29+
[Run an XTable sync on Apache Spark](/docs/how-to-spark-runtime).

0 commit comments

Comments
 (0)