feat(java): describe physical batches and writes without a format - #961
Open
keksmd wants to merge 3 commits into
Open
feat(java): describe physical batches and writes without a format#961keksmd wants to merge 3 commits into
keksmd wants to merge 3 commits into
Conversation
The reader and writer verticals both need one neutral description of a row group before either can name a file format. This adds that description: a recursive column type, a named field, an ordered schema, a row whose values follow one documented Java mapping, a finite record batch, and a closeable cursor over batches. The write half of the physical IO boundary rides along, because it is the smaller consumer of those types and shares every one of them: a write request binds a URI, a schema, and a target disposition, and a physical writer drains a cursor into that target. The module depends on nothing else in the project, so a format backend can implement it without inheriting the metadata or storage layers. Part of apache#947. Rejected: exposing Arrow as the public batch representation, which would make every consumer inherit an Arrow runtime for a contract that only needs row access. Not-tested: no format backend implements these interfaces yet; the contract is pinned against an in-memory reference implementation.
Replace the provisional row access shape with a validated vector batch so physical backends can align with Arrow without imposing its runtime on every Java consumer. Preserve indexed duplicate schema fields, model recursive nested shapes, and make append an explicit write disposition.\n\nPart of apache#959.\n\nRejected: making Apache Arrow a mandatory dependency of graphar-io-api.\n\nDirective: format adapters own vector lifetime and Java value representation.\n\nNot-tested: no physical format backend implements this boundary yet.
Contributor
Author
|
@SemyonSinchenko Спасибо, поправил все пять пунктов в 5837bd5.
Focused verify: 33 tests, 0 failures; spotless и pre-commit также прошли. |
Member
This was referenced Aug 24, 2026
keksmd
added a commit
to keksmd/incubator-graphar
that referenced
this pull request
Aug 24, 2026
Complete the core-only ordered-adjacency contract: validate offset chunks, select exact half-open edge chunk ranges, and delegate URI resolution to immutable GraphAr metadata. Relates to apache#967. Rejected: reading Parquet offsets in graphar-core; io-parquet remains behind the open apache#961 API change.
4 tasks
Make column-kind classification exhaustive and give immutable schema and write requests value semantics for deduplication and queued write planning. Relates to apache#961.
Contributor
Author
|
@SemyonSinchenko Спасибо за второй проход — сверил весь набор review-комментариев и поправил оставшиеся два minor пункта в
Предыдущие пять замечаний из первого review также остаются закрытыми: value Проверка: Когда будет минутка, пожалуйста, посмотри ещё раз и, если всё ок, approve. |
Member
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for this PR
Part of #947, tracked by #959. The reader and writer verticals both need one format-neutral description of a physical batch before either can name a file format. Without it, the Parquet backend and the GraphAr reader would each invent their own representation and the two would have to be reconciled later.
What changes are included in this PR?
A new dependency-light
graphar-io-apimodule holding a columnar physical-batch description plus the write half of the physical IO boundary:ColumnType,Field,Schema— recursive Arrow-shaped schema descriptions. NestedLIST,FIXED_SIZE_LIST,STRUCT, andMAPretain child field names/nullability; fixed binary and decimal parameters are explicit. Duplicate field names are valid because physical access is by position.ValueVector,VectorRecordBatch,BatchCursor— a batch is a schema, same-length vectors, and a row count.VectorRecordBatchrejects mismatched schema fields, vector width, and value counts before publication. It deliberately resembles Arrow's vector access shape without imposing Arrow buffers, allocators, or a transitive runtime dependency.WriteRequest,WriteMode,PhysicalWriter— a URI, a schema, a target disposition, and the writer that drains a cursor.CREATE_NEW,OVERWRITE, andAPPENDare explicit.A format adapter owns vector lifetime and the Java representation returned by
ValueVector#getObject. That lets a Parquet or Arrow adapter expose native/lifecycle-bound data when safe, while keeping the base API usable without Arrow.The read half of the boundary (
ReadRequest, projection, filter, row range, capability reporting,PhysicalReader) is deliberately left out and will follow as its own PR against this base.Are these changes tested?
Yes — 33 tests. They cover value-object equality, indexed duplicate schema fields, recursive nested types, vector/batch structural validation, nullable list elements, vector-based cursor traversal, cursor ownership, schema mismatch, and all three write dispositions against an in-memory reference writer.
Are there any user-facing changes?
A new module and package
org.apache.graphar.io. Nothing existing changes.Checklist
make cpplintbefore submitting when changed files are in thecppdirectory.pre-commit runbefore commit the changed files.