Skip to content

[MINOR] Stop TestIcebergSync generating zero-record data files - #919

Open
slachiewicz wants to merge 1 commit into
apache:mainfrom
slachiewicz:iceberg-sync-zero-record-flake
Open

[MINOR] Stop TestIcebergSync generating zero-record data files#919
slachiewicz wants to merge 1 commit into
apache:mainfrom
slachiewicz:iceberg-sync-zero-record-flake

Conversation

@slachiewicz

@slachiewicz slachiewicz commented Aug 25, 2026

Copy link
Copy Markdown
Member

TestIcebergSync fails intermittently with expected: <2> but was: <1>, most recently on #912, where the change under review can't reach the test: it mocks both the schema extractor and the partition spec extractor.

What is the purpose of the pull request

getDataFile draws recordCount from an unseeded Random over [0, 10000), so it occasionally builds a data file with no records. Iceberg's InclusiveMetricsEvaluator returns ROWS_CANNOT_MATCH for any file whose recordCount is 0, which prunes that file from every filtered scan. The tests that pass a filter expression to validateIcebergTable then see one file fewer than they staged, and the file count assertion fails.

The odds are about one run in five thousand per filtered test, which is why it reads as an unrelated red build rather than a reproducible failure.

fileSizeBytes is drawn the same way and is left alone. At roughly 10 KB a file, both files stay well under the 4 MB open file cost and the 128 MB split target, so planTasks returns a single combined scan task regardless of the draw.

Brief change log

  • TestIcebergSync#getDataFile draws recordCount from [1, 10000].

Verify this pull request

This change is a test fixture fix and can be verified as follows:

  • ./mvnw -pl xtable-core test -Dtest=TestIcebergSync runs 9 tests, all green, on JDK 11.
  • Giving one of the three staged files a recordCount of 0 and leaving the others alone reproduces the CI failure exactly: TestIcebergSync.testNestedFieldPartitioning:680->validateIcebergTable:845 expected: <2> but was: <1>. Zeroing all three instead loses every file and fails one assertion earlier, at line 843.
  • The pruning rule is the first branch of InclusiveMetricsEvaluator$MetricsEvalVisitor.eval in iceberg-api, which returns ROWS_CANNOT_MATCH when recordCount is 0, before it reads any column statistics.
  • The failure this prevents was observed on [911] Carry Delta's nested field ids into the converted schema #912, whose change mocks both extractors and so cannot affect the scan.

This change was created with AI assistance.

Iceberg's InclusiveMetricsEvaluator returns ROWS_CANNOT_MATCH for any
file whose recordCount is 0, so such a file is pruned from every
filtered scan. getDataFile drew recordCount from an unseeded Random
over [0, 10000), so roughly one run in five thousand dropped a file
that validateIcebergTable expected and failed with "expected: <2> but
was: <1>".

Do not fold this back into a plain nextInt: a data file with no records
is not a fixture any of these tests mean to build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant