Skip to content

Configure compression from the store URL - #50

Open
sduchesneau wants to merge 3 commits into
developfrom
feature/zstd-config-env
Open

sduchesneau wants to merge 3 commits into
developfrom
feature/zstd-config-env

Conversation

@sduchesneau

@sduchesneau sduchesneau commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Every store now reads two query parameters on its URL, whatever the scheme:

  • compression=zstd|gzip|none overrides the compression given to the constructor and the Compression option. The .zst/.gz suffix of the extension follows it, so NewDBinStore("gs://bucket/path?compression=gzip") writes and reads .dbin.gz. Extensions without a compression suffix (NewSimpleStore, NewStoreFromFileURL) are kept as is.
  • compression_config=<level> or <level>/<window MiB> sets the zstd encoder level and window: best, better/32, best/64. It applies to that store only, and is an error on a store that does not compress with zstd.

NewStoreFromFileURL, OpenObject and ReadObject now take the compression from the file name when no Compression option is given (.zst, .gz), so compression_config works there without compression=zstd. They used to read such files as is; ?compression=none keeps that. No caller in our repos relied on it: the firehose-core ones pass Compression("zstd") already.

ObjectURL now puts the store URL query after the object path instead of appending the path to the query.

The settings are per store, so a merger can write merged blocks with best/32 while its one-block store keeps the defaults. An invalid value makes the constructor fail. ResolveCompression is exported and holds these rules.

This bumps klauspost/compress from v1.10.2 to v1.20.1 (Go 1.25), because in v1.10.2 better and best are aliases of default.

Readers need no change for compression_config, since the window comes from the frame header, but they need memory for it. The zstd command line tool refuses windows above 128 MiB by default. Readers do need the same compression as the writer, since it changes file names.

On BNB Chain merged blocks (16-core arm64), savings for every TiB written compressed with the defaults:

config saved per TiB encode decode
default / lib-default (now) — 879 MB/s 2362 MB/s
better / lib-default 107 GiB (10.5%) 475 MB/s 2466 MB/s
better / 16MB 137 GiB (13.4%) 503 MB/s 1921 MB/s
better / 32MB 142 GiB (13.9%) 490 MB/s 1347 MB/s
better / 128MB 142 GiB (13.9%) 496 MB/s 620 MB/s
best / lib-default 135 GiB (13.2%) 73 MB/s 2512 MB/s
best / 16MB 181 GiB (17.7%) 70 MB/s 1955 MB/s
best / 32MB 229 GiB (22.4%) 66 MB/s 1365 MB/s
best / 64MB 270 GiB (26.4%) 64 MB/s 892 MB/s
best / 128MB 293 GiB (28.6%) 64 MB/s 623 MB/s

Bumps klauspost/compress to v1.20.1: in v1.10.2 the better and best levels were aliases of default.

@maoueh maoueh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving nonethless as this make it easy to experiment for some elements.

Comment thread stores.go Outdated
// Valid `compressionType` values:
// - <empty> No compression
// - zstd Use ZSTD compression
// - zstd Use ZSTD compression, tuned with the DSTORE_ZSTD_CONFIG env var

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An env var applies bindly to everything, uncertain if it's the default best path. Options available at store creation, ideally via DSN seems like a better way to tweak this on a store per store basis.

Replaces DSTORE_ZSTD_CONFIG, which applied to every zstd store in the
process, with compression and compression_config URL parameters.
@sduchesneau sduchesneau changed the title Configure zstd level and window with DSTORE_ZSTD_CONFIG Configure compression from the store URL Oct 6, 2026
Also keep the store URL query after the object path in ObjectURL.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants