feat(java): add an optional S3 storage adapter - #966
Open
keksmd wants to merge 2 commits into
Open
Conversation
S3 range reads and staged writes need to satisfy the storage API without leaking AWS dependencies into metadata or core modules. Relates to apache#965. Rejected: a mandatory S3 or Hadoop dependency in the storage API. Not-tested: a real MinIO endpoint; the follow-up opt-in transport test remains separate.
Contributor
Author
|
Hi @SemyonSinchenko — this is the next independent Java slice after #960/#962: an optional S3 adapter over the now-merged storage API. It is intentionally isolated from #961 and has credential-free range/write contract coverage. I would appreciate your review when you have time. |
Exercise staged conditional publication and range reads against an actual S3-compatible endpoint, while keeping the test opt-in for normal SDK verification. Relates to apache#965.
Contributor
Author
|
Updated this PR into the complete S3 transport vertical: it now includes the opt-in real-MinIO gate (staged conditional publication and range reads), while normal CI stays credential- and Docker-free. |
SemyonSinchenko
self-requested a review
August 24, 2026 14:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for this PR
Implements #965: an optional, dependency-isolated Java S3 storage adapter over the merged storage API.
What changes are included in this PR?
graphar-storage-s3, backed by caller-owned AWS SDK v2S3Client.s3://locations, uses HeadObject/HTTP Range GET for independent seekable reads, and stages sequential output locally before conditional publication.The production code does not start, configure, or close an S3 client. The MinIO test is intentionally opt-in: normal SDK CI has no Docker or credential requirement.
Are these changes tested?
Yes.
mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am clean verify -Dspotless.check.skip=true GRAPHAR_MINIO_ENDPOINT=http://localhost:19000 mvn --no-transfer-progress -f maven-projects/pom.xml -pl storage-s3 -am -Dtest=MinioS3StorageIntegrationIT -Dsurefire.failIfNoSpecifiedTests=false testBoth pass locally, along with Spotless, pre-commit, and
git diff --check.Are there any user-facing changes?
Yes. Java users can construct
S3Storagewith their configured AWS SDK v2 client and access GraphArs3://data through the storage API.Checklist
make cpplint(if this PR contains C++ changes)pre-commit