Skip to content

Commit f2d9873

Browse files
vinishjail97claude
andauthored
[836] xtable-spark-runtime: thin drop-in Spark bundle for 0.4.x (Spark 3.4 + 3.5) (#843)
* [836] xtable-spark-runtime: thin drop-in Spark bundle (Hudi 0.x) Ports the xtable-spark-runtime packaging slice onto the Hudi 0.x line (main-hudi-0x, Hudi 0.14 / Spark 3.4). New thin, relocated bundle that runs an incremental XTable sync inside a Spark job; engines are provided by the cluster, never bundled. Module: - xtable-spark-runtime_${scala.binary.version}: xtable-core compile; Spark/Hadoop and the engines (Hudi/Iceberg/Delta) provided. Curated shade allowlist (xtable modules + guava/protobuf/commons-cli relocated); avro/parquet/jackson NOT relocated (exchanged with the engines). Thin ~3.7 MB bundle. - XTableSparkSync: standalone spark-submit entry point (Apache Commons CLI). - XTableSyncService / TableSyncSpec: build an INCREMENTAL ConversionConfig and run ConversionController.sync; target metadata path = source data path (required by Hudi; Iceberg data lives under <basePath>/data). Hudi 0.14 specifics (vs the Hudi 1.x variant on main): - No hudi-hadoop-common (that split is 1.x); hudi-common provides FSUtils. - Engine classpath uses the hudi-spark bundle (its regenerated Avro model classes link on Avro 1.12, which Iceberg 1.9.2 requires) plus hudi-java-client for the Hudi target's Java write client, with the raw hudi-common excluded so the bundle's clean DecimalWrapper wins. ConversionTargetFactory: make ServiceLoader discovery resilient so a subset of engines works when others are absent (warn + skip on LinkageError/ServiceConfigurationError; name-based Delta-Kernel check). ITXTableSparkRuntimeBundle: spark-submits the shaded jar for one case per direction across Hudi, Iceberg and Delta (source and target), engines on a flat classpath, asserting data-equivalence over comparable scalar columns. Requires a Spark 3.4 SPARK_HOME; skipped otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [836] xtable-spark-runtime: address review feedback (packaging, licensing, RFC) Publish the shaded jar as the MAIN artifact with a dependency-reduced POM (shadedArtifactAttached=false, createDependencyReducedPom=true), matching iceberg-spark-runtime / hudi-spark-bundle. Resolving the coordinate via a Maven dependency or --packages now yields the relocated bundle and pulls no un-relocated transitive deps; --jars is equivalent. The thin main jar + separate -bundle classifier (which re-introduced the cluster guava clash) is gone. IT findBundleJar() now picks the shaded main jar. Pass release/scripts/validate_shaded_license_coverage.sh (the existing allowlist gate): the shade <includes> must equal the runtime dependency tree, so jackson / scala-library / log4j-1.2-api - which every Spark runtime supplies - are declared provided (dropped from the tree, kept off the shaded jar) and excluded from the IT's flat engine classpath so Spark's own copies win. avro/parquet stay on the flat classpath (the engine's newer avro must win over Spark 3.4's). Bundled-dependency licensing: add META-INF/LICENSE-bundled and NOTICE-bundled (wired via IncludeResourceTransformer) attributing the only bundled third-party - guava's closure (Apache-2.0) and commons-cli (Apache-2.0), plus checker-qual (MIT). Remove the dead protobuf-java allowlist entry and relocation: protobuf resolves as provided and was never bundled. RFC-3: scope v1 to the CLI (XTableSparkSync); mark the config-only XTableSyncListener on-ramp (and XTableSparkConfig / PlanTargetResolver) as a deferred follow-up. Fix the activation example to the shipped CLI and document the required engine Avro version + flat-classpath placement. XTableSyncService: normalize sourceFormat with Locale.ROOT. Root pom: exclude the shade-generated dependency-reduced-pom.xml from spotless (no license header; CI runs clean install). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [836] xtable-spark-runtime: trim duplicated pom comments Consolidate repeated rationale in the bundle pom (no functional change): - Tell the hudi-common DecimalWrapper / Avro-1.8-1.9 story once (on the hudi-spark bundle dep); the hudi-java-client exclusion just points to it. - State "engine Avro must win on a flat classpath" once (engine-classpath plugin); the avro dep and excludeGroupIds comments reference it. - Explain jackson/scala-library are Spark-supplied once. - Fix a stale "Spark 3.5" reference to Spark 3.4 (this is the Hudi 0.x line). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [836] ci: trigger Maven CI + License Check on main-hudi-0x Add main-hudi-0x to the push and pull_request branch filters of both workflows (same change as #841) so CI runs on PRs targeting the Hudi 0.x release line, including this one. A pull_request workflow's branch filter takes effect from the PR's own branch, so this makes CI fire on #843. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review feedback: guard ServiceLoader.hasNext() and drain spark-submit stdout off-thread - ConversionTargetFactory: hasNext() resolves provider classes lazily and can throw ServiceConfigurationError too, so move it inside the existing ServiceConfigurationError|LinkageError guard alongside next(). - ITXTableSparkRuntimeBundle: drain spark-submit stdout on a background thread so a hung process is caught by waitFor(10min) instead of blocking forever on readOutput() reading until EOF. * [836] ci: point Maven CI + License Check triggers at branch-0.4 main-hudi-0x was deleted and replaced by branch-0.4 (the 0.4.0 release line), so update the push/pull_request branch filters accordingly. * [836] ci: validate xtable-spark-runtime bundle against a real Spark distro ITXTableSparkRuntimeBundle spark-submits the shaded bundle jar and self-skips unless SPARK_HOME is set, so the main CI never runs it. Add a path-filtered workflow that installs a matching Spark 3.4 distribution from the Apache archive, sets SPARK_HOME, and runs the failsafe IT. * [836] ci: run spark-runtime validation on every branch PR (drop path filter) The workflow is intended to be a required status check on branch-0.4. A required check whose workflow is skipped by a path filter never reports, leaving PRs blocked on a pending check, so run it unconditionally on the covered branches. The Spark distro is cached, so the added cost is the reactor build plus the ~20s IT. * [836] xtable-spark-runtime: address review feedback (CLI validation, unix flags, docs) - XTableSparkSync: validate --sourceformat/--targets up front and fail fast (before SparkSession creation) on empty or unsupported values. Split the allowed sets: sources are Hudi/Iceberg/Delta/Paimon/Parquet, targets are Hudi/Iceberg/Delta (Paimon/Parquet are read-only, no ConversionTarget). - XTableSyncService.sourceProviderFor: wire Paimon and Parquet source providers (previously threw UnsupportedOperationException); engines remain cluster-provided. - Rename CLI long-opts to unix-style lowercase (--basepath, --sourceformat, --tablename, --datapath, --partitionspec); update javadoc, IT and RFC example. - basePathToName -> basePathToTableName: handle "/", trailing slashes and null by throwing with a "pass --tablename" hint instead of an empty table name. - Add XTableSparkSyncTest covering table-name derivation and source/target format validation. - ConversionTargetFactory: log the skipped provider's error class/message so operators can distinguish an intentionally-absent engine from a linkage error. - spark-runtime-validation.yml: add a workflow_dispatch spark_version input and document why the Spark 3.4 line is pinned (Delta 2.4.0 is Spark-3.4-only). - RFC-3: add a supported-formats/engine-versions section, "from application code" (Scala/Java + PySpark) activation examples, and list @vinothchandar as an approver. * [836] xtable-spark-runtime: support Spark 3.5 via Delta Kernel Delta source/target now auto-switch from delta-core to the Spark-free Delta Kernel implementation on Spark 3.5+ (where the bundled delta-core does not run), controlled by a single --usedeltakernel toggle that is auto-enabled by Spark version. Hudi/Iceberg sync was already Spark-free. Also runs the spark-runtime bundle IT on both Spark 3.4.3 and 3.5.9 via a CI matrix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [836] Give Maven CI Build and License Check unique job names Both workflows used a job named "build", so both reported the same status-check context and could not be required distinctly. Set unique job names so branch protection (see #848) can require each on branch-0.4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [836] xtable-spark-runtime: add --datasetconfig for multi-table sync XTableSparkSync accepts a --datasetconfig YAML (sourceFormat, targetFormats, datasets[]) to sync multiple tables in one run, mutually exclusive with the single-table --basepath/--sourceformat/--targets flags. The config is read through the Spark Hadoop config, so it may live on a local or cloud (s3/gcs/abfs) path, and reuses the same schema as the RunSync utility. Parsed with SnakeYAML's SafeConstructor (plain maps/lists only, no arbitrary type instantiation). SnakeYAML is bundled and relocated to org.apache.xtable.shaded (like guava/commons-cli) because Spark ships its own version (1.33 on 3.4, 2.0 on 3.5) that would otherwise clash; its multi-release classes are filtered out so no un-relocated org.yaml classes remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7e273e6 commit f2d9873

14 files changed

Lines changed: 2240 additions & 3 deletions

File tree

.github/workflows/mvn-ci-build.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ on:
3333

3434
jobs:
3535
build:
36+
name: Maven CI Build
3637
runs-on: ubuntu-latest
3738
timeout-minutes: 30
3839
steps:

.github/workflows/mvn-license-check.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ on:
3333

3434
jobs:
3535
build:
36+
name: License Check
3637
runs-on: ubuntu-latest
3738
steps:
3839
- uses: actions/checkout@v6
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
name: Spark Runtime Bundle Validation
18+
19+
# Validates the shaded xtable-spark-runtime bundle end-to-end: ITXTableSparkRuntimeBundle
20+
# spark-submits the bundled jar against a real Spark distribution for one case per direction
21+
# across Hudi, Iceberg and Delta. The IT self-skips unless SPARK_HOME is set, so the main CI
22+
# never exercises it; this workflow installs each Spark distro in the matrix and runs it.
23+
# Runs on both Spark 3.4 and 3.5: Hudi/Iceberg sync is Spark-free, and a Delta target/source
24+
# auto-switches from delta-core to the Spark-free Delta Kernel on Spark 3.5+.
25+
26+
concurrency:
27+
group: ${{ github.workflow }}-${{ github.ref }}
28+
cancel-in-progress: ${{ !contains(github.ref, 'main') }}
29+
30+
# Runs on every push/PR to the covered branches (no path filter) so the check always
31+
# reports — required status checks that never trigger leave a PR blocked on a pending check.
32+
on:
33+
push:
34+
branches:
35+
- "main"
36+
- "branch-0.4"
37+
38+
pull_request:
39+
branches:
40+
- "main"
41+
- "branch-0.4"
42+
43+
workflow_dispatch:
44+
45+
env:
46+
SPARK_HADOOP_PROFILE: "hadoop3"
47+
48+
jobs:
49+
validate-spark-runtime-bundle:
50+
runs-on: ubuntu-latest
51+
timeout-minutes: 30
52+
strategy:
53+
fail-fast: false
54+
matrix:
55+
# 3.4: Delta via delta-core. 3.5: Delta auto-switches to the Spark-free Delta Kernel.
56+
spark_version: ["3.4.3", "3.5.9"]
57+
env:
58+
SPARK_VERSION: ${{ matrix.spark_version }}
59+
steps:
60+
- uses: actions/checkout@v6
61+
62+
- name: Set up JDK 11
63+
uses: actions/setup-java@v5
64+
with:
65+
java-version: '11'
66+
distribution: 'temurin'
67+
cache: maven
68+
69+
- name: Cache Spark distribution
70+
uses: actions/cache@v4
71+
with:
72+
path: ~/spark
73+
key: spark-${{ env.SPARK_VERSION }}-bin-${{ env.SPARK_HADOOP_PROFILE }}
74+
75+
- name: Install Spark
76+
run: |
77+
set -euo pipefail
78+
SPARK_DIR="spark-${SPARK_VERSION}-bin-${SPARK_HADOOP_PROFILE}"
79+
if [ ! -d "$HOME/spark/$SPARK_DIR" ]; then
80+
mkdir -p "$HOME/spark"
81+
echo "Downloading $SPARK_DIR from the Apache archive"
82+
curl -fsSL "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_DIR}.tgz" \
83+
| tar -xz -C "$HOME/spark"
84+
fi
85+
echo "SPARK_HOME=$HOME/spark/$SPARK_DIR" >> "$GITHUB_ENV"
86+
87+
- name: Build reactor (skip tests)
88+
run: ./mvnw clean install -ntp -B -DskipTests
89+
90+
- name: Run xtable-spark-runtime bundle IT
91+
env:
92+
SPARK_LOCAL_IP: "127.0.0.1"
93+
run: ./mvnw verify -ntp -B -pl xtable-spark-runtime

pom.xml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
<module>xtable-aws</module>
5555
<module>xtable-hive-metastore</module>
5656
<module>xtable-service</module>
57+
<module>xtable-spark-runtime</module>
5758
</modules>
5859

5960
<properties>
@@ -937,6 +938,8 @@
937938
<excludes>
938939
<exclude>.idea/**</exclude>
939940
<exclude>${target.dir.pattern}</exclude>
941+
<!-- Generated by maven-shade-plugin (no license header); not source. -->
942+
<exclude>**/dependency-reduced-pom.xml</exclude>
940943
<exclude>website/node_modules/**</exclude>
941944
<exclude>website/build/**</exclude>
942945
<exclude>website/.docusaurus/**</exclude>

0 commit comments

Comments
 (0)