Set delta kernel as the default path for 0.5.0 release

[vinishjail97]

Feature Request / Improvement

The Delta Kernel conversion source and target are on main (#729, #801), but they are opt-in. A Delta sync only routes through them when the target table sets xtable.delta.target.use_kernel=true, which DeltaConversionTargetConfig reads and ConversionTargetFactory dispatches on. The default is false, so the default Delta path today is still Delta Standalone.

This issue tracks making Kernel the default for the 0.5.0 release, together with the work that has to land before that flip is safe.

Kernel matters beyond parity. It removes the last hard Spark dependency in the sync path, since the Hudi and Iceberg paths are already pure Java and Delta was the one format anchoring us to delta-spark and DeltaLog.

Current state

• xtable.delta.target.use_kernel defaults to false. Both implementations are registered under TableFormat.DELTA in the same META-INF/services file, and ConversionTargetFactory picks between them on that flag.
• The Kernel path is covered only by its own suites: ITDeltaKernelConversionSource, TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync, plus the extractor, partition, schema and stats unit tests.
• ITConversionController is where every other source and target combination is validated for dataset equivalence across sync modes and partitioning. It has no Kernel coverage. Its generateTestParametersForFormatsSyncModesAndPartitioning() iterates HUDI, DELTA, ICEBERG, PAIMON, and getConversionSourceProvider(String) returns DeltaConversionSourceProvider for DELTA with no Kernel branch. DeltaKernelConversionSourceProvider is not even imported there.

Prerequisites, before the default changes

1. Cover Kernel in ITConversionController, on both sides:

   • as a source, so DeltaKernelConversionSourceProvider runs the same equivalence checks DeltaConversionSourceProvider runs today, across both sync modes and both partitioning cases;
   • as a target, by setting xtable.delta.target.use_kernel=true for the Kernel variant, so both writers are validated.

   Keep the Standalone cases as they are. The goal is both implementations passing the same assertions, not replacing one with the other. If doubling the whole matrix costs too much wall-clock time, a separate parameter or a targeted subset is fine, as long as both sync modes and both partitioning cases are covered for each of source and target.

2. Re-verify the Delta conversion issues reported recently against the Kernel path, so the flip does not quietly regress fixes that only exist on the Standalone path: XTable performs incorrect schema sync during Delta to Iceberg conversion when a column is renamed in the Delta table. #711 (column rename during schema sync), Handle log truncation in Delta to Iceberg incremental sync #779 (log truncation on Delta to Iceberg incremental sync), _delta_log folder to be considered as partition when checkpoint parquet files present by hudi reader without metadata table #813 (_delta_log treated as a partition), Delta incremental sync reloads the table snapshot on every commit #860 (snapshot reloaded on every commit), Issue with Metadata Reconciliation Between Iceberg and Delta Tables During Snapshot Updates #586, NullPointerException when using Snowflake Catalog as source and Delta Lake as target #641.

3. Answer feature parity for deletion vectors (Verify if delta kernel can be used for delta conversion source and targets #713). If Kernel cannot express something Standalone can, that needs to be known and documented before it becomes the default rather than after.

The change itself

• Flip the default of xtable.delta.target.use_kernel to true.
• Keep the flag as an escape hatch for one release, so a user who hits a regression can go back to Standalone without downgrading.
• Call the change out in the release notes, with the flag documented as the way to opt out.

Out of scope

The packaging and module questions raised on the same dev@ thread. The Kernel path being solid is a precondition there too, but the packaging decision is independent of this issue.

Context from the dev@ discussion, where the point was made that the dedicated suites are not the right bar and the shared harness is:
https://lists.apache.org/thread/bx1xyh0ml30r60od1hdvhndyd16b375j

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[vinishjail97]

@vaibhavk1992 Can you pick this up as you have worked on delta kernel? I'm tracking this as part of 0.5.0 release.

[vaibhavk1992]

Sure @vinishjail97 , please assign this to me.

[vinishjail97]

@vaibhavk1992 You can start working on it..Github is not allowing to assign, will check why.