Repository navigation
Configure compression from the store URL - #50
Open
sduchesneau wants to merge 3 commits into
Open
sduchesneau wants to merge 3 commits into
sduchesneau wants to merge 3 commits into
Conversation
Bumps klauspost/compress to v1.20.1: in v1.10.2 the better and best levels were aliases of default.
sduchesneau
marked this pull request as ready for review
October 6, 2026 17:27
This was referenced Oct 6, 2026
maoueh
approved these changes
Oct 6, 2026
maoueh
left a comment
Contributor
There was a problem hiding this comment.
Approving nonethless as this make it easy to experiment for some elements.
| // Valid `compressionType` values: | ||
| // - <empty> No compression | ||
| // - zstd Use ZSTD compression | ||
| // - zstd Use ZSTD compression, tuned with the DSTORE_ZSTD_CONFIG env var |
Contributor
There was a problem hiding this comment.
An env var applies bindly to everything, uncertain if it's the default best path. Options available at store creation, ideally via DSN seems like a better way to tweak this on a store per store basis.
Replaces DSTORE_ZSTD_CONFIG, which applied to every zstd store in the process, with compression and compression_config URL parameters.
Also keep the store URL query after the object path in ObjectURL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every store now reads two query parameters on its URL, whatever the scheme:
compression=zstd|gzip|noneoverrides the compression given to the constructor and theCompressionoption. The.zst/.gzsuffix of the extension follows it, soNewDBinStore("gs://bucket/path?compression=gzip")writes and reads.dbin.gz. Extensions without a compression suffix (NewSimpleStore,NewStoreFromFileURL) are kept as is.compression_config=<level>or<level>/<window MiB>sets the zstd encoder level and window:best,better/32,best/64. It applies to that store only, and is an error on a store that does not compress with zstd.NewStoreFromFileURL,OpenObjectandReadObjectnow take the compression from the file name when noCompressionoption is given (.zst,.gz), socompression_configworks there withoutcompression=zstd. They used to read such files as is;?compression=nonekeeps that. No caller in our repos relied on it: the firehose-core ones passCompression("zstd")already.ObjectURLnow puts the store URL query after the object path instead of appending the path to the query.The settings are per store, so a merger can write merged blocks with
best/32while its one-block store keeps the defaults. An invalid value makes the constructor fail.ResolveCompressionis exported and holds these rules.This bumps
klauspost/compressfrom v1.10.2 to v1.20.1 (Go 1.25), because in v1.10.2betterandbestare aliases ofdefault.Readers need no change for
compression_config, since the window comes from the frame header, but they need memory for it. Thezstdcommand line tool refuses windows above 128 MiB by default. Readers do need the samecompressionas the writer, since it changes file names.On BNB Chain merged blocks (16-core arm64), savings for every TiB written compressed with the defaults: