Side-car JSON does not scale: 5.5x larger than the index at 100k docs, and save peaks at 2x the payload

[RyanCodrai]

Split out of #350, whose four enumerated defects are all fixed (items 1 and 2 in b101343, items 3 and 4 in #423). This is the one observation from that issue that remains open, and #350 itself labels it "Not a bug at current targets, but it's the first thing that bites at 10M" — recording it separately so #350 can close without losing it.

Measured at 100k docs, 4-field metadata, dim 64:

artifact	size
JSON side-car	24.0 MB
.tvim index	4.4 MB

The side-car is 5.5x larger than the index it accompanies. Load takes 0.3 s at 472 MB peak RSS.

Extrapolating to 1M docs: ~240 MB side-car, ~3 s load, multi-GB peak.

The peak is the sharper end of it. json.dumps materialises the entire payload as a second in-memory string before anything is written, so save() peaks at roughly 2x the payload — the dict and its serialised form are both resident at once. json.load on the read side has the same shape.

Nothing here is wrong at the sizes the integrations target today, and the format is deliberately plain inspectable JSON (documented as "plain JSON, never pickle"), which is worth keeping. Possible directions, none obviously right:

• json.dump(payload, f) instead of f.write(json.dumps(payload)) — streams the encoder's chunks straight to the file and removes the 2x save peak without changing the format at all. Cheapest change by a wide margin. Note it interacts with the atomic-write path: the temp file is currently written from a single string, and atomic_save deliberately serialises before touching any file so a bad payload fails clean (see _check_json_faithful) — streaming would need to keep that ordering property, e.g. by validating first (which it already does) and accepting that a mid-write failure leaves a temp file, which the finally already unlinks.
• A streaming reader for load, or a line-delimited payload section, to avoid materialising the whole dict.
• Splitting bulk metadata out of the side-car entirely.

The handle/keyset consistency checks (check_persisted_handles, check_sidecar_keysets) must keep working under whatever shape this takes.

🤖 Generated with Claude Code

[github-actions]

Claude encountered an error after 1s —— View job

---

I'll analyze this and get back to you.

[RyanCodrai]

Downstream of #427: shrinking the side-car to ids + metadata removes the document text that makes it 5.5x the index. Recommend deciding #427 first rather than optimising a structure that may largely go away. Labelled for your decision.

[RyanCodrai]

/summary

[tq-bot]

Plain-English summary: When turbovec saves a search index to disk, it writes a companion file alongside it that holds things like document text and metadata. This issue points out that the companion file grows much faster than the index itself — at 100,000 documents it's already about five and a half times bigger — and that saving it temporarily uses roughly twice its size in memory, because the whole thing is built in memory before any of it is written out. Loading has the same problem in reverse. None of this causes trouble at the sizes the project supports today, but it would become the first real bottleneck for anyone pushing toward millions of documents.

The issue lists a few possible fixes, from a cheap change that writes the file gradually instead of all at once, to restructuring what gets stored in the companion file. The maintainer has since noted that a separate open question (#427) may remove most of the bulky content from this file anyway, so the sensible order is to settle that decision first rather than optimize something that might largely disappear. This issue is on hold awaiting that call.