Repository navigation
Fix listing bounds in S3 and GS stores - #48
Merged
Merged
Conversation
WalkFromTo compared its exclusive end point against a prefix-stripped name on S3, and GS lost the trailing slash of that bound, so a folder end point skipped the object named after it. Ranged folder listings now stop at the upper bound instead of listing the rest of the prefix.
sduchesneau
commented
Aug 25, 2026
sduchesneau
left a comment
Contributor
Author
There was a problem hiding this comment.
LGTM (claude/gpt sol wrote that)
UlysseCorbeil
approved these changes
Aug 25, 2026
UlysseCorbeil
left a comment
Contributor
There was a problem hiding this comment.
Nothing jumps out, lgtm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bugs
Both were caught by new shared
storetestscases, verified by reverting the source and watching each one fail on exactly the store that had it (MinIO / fake-gcs).WalkFromTocompared its exclusive end point against a prefix-stripped name while the walked names are store-relative. With a non-empty prefix the bound never matched, so the walk yielded keys past it and paged to the end of the prefix.WalkFromTobuilt the listing's end offset withfilepath.Join, which ate the trailing/. Walking up toalpha/dropped the object namedalpha, which sorts before the bound.Same results, less work
ListFoldersFromToon S3 and Azure returns at the first folder>= exclusiveToinstead of listing the rest of the prefix to discard it. Azure has no server-side bound, so this is its only lever.ListFoldersFromTopushes its lower bound down whole, minus the trailing/that makesStartAfterexclusive. It used to drop the bound's last character, and push nothing at all when a single character was left.WalkFromTostarts at the key right before the starting point (helloworld.html→helloworld.htmk) instead of the stem (helloworld.htm), which had the service send every key in between for the walk to filter out. On fixed-width keys such as block numbers that window held every key sharing the stem; it now holds none. NewkeyBefore()decrements the last rune, so the marker stays valid UTF-8, and falls back to dropping it (control characters, invalid bytes, the surrogate boundary).Tests
storetests:TestWalkFromTo_WithPrefix,TestWalkFromTo_FolderEndPoint,TestListFoldersFromTo_SingleCharacterBounds— the suite only ever walked ranges with an empty prefix and never used a slash-terminated end point, which is how both bugs got in.s3store_mock_test.go: a recordingRoundTripperserving canned pages, asserting thestart-after/prefix/delimitereach bound shape produces and proving early termination by round-trip count — a real backend can't show you that. Plus akeyBeforetable with a sorts-before invariant.gsstore_test.go:TestGSStore_calculateEndOffset.Green against MinIO, Ceph, fake-gcs and Azurite (
docker compose up -d), plus local/memory/mock.Unrelated, pre-existing:
azure_test.goTestAzureStoreWriteObjectgates onAZURE_STORAGE_KEYbut hardcodes accountstreamingfasttest01/containermyblobs, so pointing it at Azurite 404s. Left alone.🤖 Generated with Claude Code
https://claude.ai/code/session_01ESMxv9zsuiAXikNs1wwYup