Skip to content

Copy S3 objects server-side - #49

Merged
sduchesneau merged 3 commits into
developfrom
feature/s3-server-side-copy
Sep 11, 2026
Merged

sduchesneau merged 3 commits into
developfrom
feature/s3-server-side-copy

Conversation

@sduchesneau

Copy link
Copy Markdown
Contributor

S3Store.CopyObject used to download the object and upload it back, moving every byte through the client and, on a compressed store, decompressing and recompressing it on the way. It now asks the service to do the copy: a single CopyObject call for objects up to 5 GiB, the largest S3 accepts in one call, and a multipart copy of 1 GiB parts above that, aborted if any part fails. GSStore.CopyObject already worked this way.

Verified against a local MinIO (docker compose up -d minio): write, copy, read back on both paths, plus a missing source, with the threshold and part size lowered to reach the multipart path. There are also transport-level tests, running with no backend, over the copy source, the byte ranges and the storage class. go build ./..., go vet ./... and go test ./... all pass.

The old copy downloaded the object and uploaded it back, moving every
byte through the client and recompressing it on the way.
Escaping them whole is accepted by AWS and MinIO but not by every
S3-compatible backend. A backend with no copy operation at all now falls
back to copying through the client.
@sduchesneau

Copy link
Copy Markdown
Contributor Author

Two follow-ups pushed:

The copy source now keeps its slashes literal and escapes each key segment on its own (bucket/root/a%20folder/an%20object). Escaping the whole bucket/key gave %2F separators, which AWS and MinIO decode but other S3-compatible backends do not.

A backend that answers NotImplemented or MethodNotAllowed to the copy — the single call, the multipart start, a part copy or the completion — now falls back to reading and writing the object through the client, aborting the multipart upload first. Every other error, HeadObject failures and access denied included, is still returned as is.

Verified again against MinIO, plus transport-level tests covering the escaping, both fallbacks and the non-fallback error path.

@sduchesneau
sduchesneau merged commit 3b0685e into develop Sep 11, 2026
3 checks passed
@sduchesneau
sduchesneau deleted the feature/s3-server-side-copy branch September 11, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants