Integrations should not own document text: shrink the side-car to ids + metadata

[RyanCodrai]

Summary

The four integration wrappers persist a JSON side-car holding the full document/node text alongside the .tvim index. turbovec should own indexing and search; the frameworks should own documents. The side-car should shrink to ids + metadata only, and the text should go.

This is the structural fix behind #425 (side-car 5.5x larger than the index at 100k docs) — text is what makes it large.

Why this is safe: all four frameworks support resolving text separately

Validated empirically, not from documentation. Wheel built with maturin build --release, installed non-editable into a fresh Python 3.11.12 venv; framework versions langchain-core 1.5.3 / langchain-classic 1.0.8, llama-index-core 0.14.23, haystack-ai 3.0.0, agno 2.8.6.

Framework	Resolution step	Status
LangChain	MultiVectorRetriever takes a docstore and calls docstore.mget(ids) on our hits	first-class, works today
LlamaIndex	StorageContext.docstore owns text; SimpleVectorStore itself sets stores_text = False	first-class
Haystack	AutoMergingRetriever / SentenceWindowRetriever accept a different document_store than the one that searched	first-class
Agno	none — user subclasses and overrides search	hand-rolled

Confirmed running: MultiVectorRetriever(vectorstore=TurboQuantVectorStore(...), docstore=InMemoryStore(), id_key="doc_id") retrieves documents that provably came from the docstore; LocalFileStore likewise. AutoMergingRetriever(document_store=parent_store).run(documents=turbovec_hits) returned the parent's full text while every document held in turbovec had content=None.

What has to change

Deleting the JSON writer alone is not enough — three read paths reconstruct results from the side-car, so the in-memory duplication would simply remain undisk'd:

1. LlamaIndex — query() hard-requires its own copy of the node: metadata_dict_to_node(data["node_dict"]) at turbovec-python/python/turbovec/llama_index.py:544, called from :848. Blanking the payload raises ValueError: Node content not found in metadata dict. Needs add() to stop storing node payloads and query() to return VectorStoreQueryResult(nodes=None, ids=..., similarities=...), matching SimpleVectorStore. A ~40-line prototype over turbovec.IdMapIndex was verified end to end: no vector-store JSON written at all, retrieval returns full text with the correct top hit.

   • Accepted cost: VectorStoreIndex.from_vector_store(vs) raises ValueError: Cannot initialize from a vector store that does not store text. This is by design and is what SimpleVectorStore does too.
   • Note the shipped class already accepts stores_text=False as a constructor arg, but still writes default__vector_store.nodes.json (2862 B) containing the text — so text is duplicated on disk today even when the caller opts out.

2. Agno — correctness bug, independent of this work. TurboQuantVectorDb._dedup_by_content hashes md5(doc.content). With blank content a search(limit=3) silently returns 1 result. Should dedup on id, not content.

3. LangChain — no vector-in door. add_texts embeds exactly what it stores, and add_embeddings does not exist (sorted(m for m in dir(vs) if m.startswith("add")) == ['add_documents', 'add_texts']). Passing page_content="" does not raise — it goes semantically dead: all three test documents embedded identically, every score exactly 0.2286243885755539, wrong document ranked first. Adding add_embeddings(embeddings, ids, metadatas) makes text-free a supported path rather than a trick.

4. Haystack — no change required. write_documents accepts content=None, and embedding_retrieval ranked correctly with all content blank (p1 1.004 vs p2 0.1243).

What stays

Metadata stays in the side-car. LangChain's dict and callable filters and Haystack's filter_documents both walk the local document table, so metadata cannot leave without breaking filtering. Metadata is small, and it is arguably index-adjacent anyway.

Also worth capturing when this is done: turbovec derives its own ids (_derive_doc_id = md5(f"{base_id}_{content_hash}")), so a caller-supplied id like p1 becomes 225df170ad3940d474744dbc0de92068. Any external store must key on doc.name or metadata rather than the id the user passed in — that needs documenting. Separately, Agno's content_hash_exists() is in-memory only, so with no durable turbovec state the skip-if-already-ingested check always reports "not present" and re-ingests every run; that state needs a home.

Scope

Breaking only for callers that rely on turbovec handing back text it stored. Not a major version.

[github-actions]

Claude encountered an error after 1s —— View job

---

I'll analyze this and get back to you.

[RyanCodrai]

Set aside for a maintainer decision: this is a design/restructure call, not a defect with a single correct fix. Flagged during an automated issue-fixing pass on 2026-07-30 rather than patched, because point fixes here risk entrenching the structure the issue is questioning.

[RyanCodrai]

/summary

[tq-bot]

Plain-English summary: When turbovec is used through one of the four framework integrations (LangChain, LlamaIndex, Haystack, Agno), it currently saves a full copy of every document's text in a JSON file next to its search index. That copy is redundant — the frameworks already have their own places to keep documents — and it's what makes that file balloon to several times the size of the index itself (#425). This issue proposes that turbovec stop storing text entirely and keep only document ids and metadata, with each framework's own document store supplying the text when results come back. The author tested all four frameworks and confirmed they can each resolve text on their own, so the change is safe in principle.

Simply deleting the file isn't enough, though, because parts of the code read text back out of it. The LlamaIndex integration would fail outright without its stored copy, the Agno integration has a separate bug where it accidentally collapses results that have no text, and LangChain would need a new way to accept precomputed embeddings so callers aren't forced to hand over text. Haystack needs no changes. Metadata has to stay because filtering depends on it. The change would only break users who rely on turbovec returning the text it stored, and the issue has been set aside for a maintainer to decide, since it's a design restructuring rather than a straightforward bug fix.

[RyanCodrai]

@q-turbovec-bot please investigate whether this is possible. Ideally people will be able to use the 4 integrations as a build your own setup where people can choose turboquant as the index backend and the rest is handled by an existing functionality. Let me know to what degree this is possible

[tq-bot]

Claude finished @RyanCodrai's task in 2m 21s —— View job

---

Todos

• Gather context (integration wrappers, shared persistence module)
• Assess LangChain path (MultiVectorRetriever + external docstore)
• Assess LlamaIndex path (StorageContext docstore, stores_text=False)
• Assess Haystack path (separate retrieval document_store)
• Assess Agno path
• Cross-cutting blockers (id derivation, filters, dedup, re-ingest state)
• Write up degree-of-possibility verdict

Verdict: possible for all four, but to very different degrees today

I read all four wrappers end to end (langchain.py, llama_index.py, haystack.py, agno.py) plus _persist.py, and code-verified every claim in the issue body. The short answer: the "turboquant as the index backend, framework owns the documents" composition is architecturally sound for all four frameworks, works out of the box for two of them (Haystack fully, LangChain with a caveat), needs a moderate wrapper change for LlamaIndex, and needs both a bug fix and a hand-rolled pattern for Agno.

Framework	Works today?	What's needed for a clean "bring your own docstore" story
Haystack	✅ Fully	Documentation only
LangChain	⚠️ Yes, but text is still embedded and stored by us	Small additive API: add_embeddings()
LlamaIndex	⚠️ Functionally yes; text still duplicated on disk	Moderate change: honor stores_text=False in add()/query()/persist()
Agno	❌ Not via composition	Bug fix (dedup by id, not content) + a documented subclass recipe

Haystack — works today, zero changes

write_documents only requires doc.embedding to be non-None (haystack.py:265-270); content=None passes straight through and _reconstruct (haystack.py:1020) returns it as-is. Haystack's retrievers (AutoMergingRetriever, SentenceWindowRetriever) accept a different document_store than the one that searched, so the pattern "content-free documents + embeddings into TurboQuantDocumentStore, full text in an InMemoryDocumentStore" is first-class framework functionality. The issue's empirical result (correct ranking with all content blank) is consistent with the code: ranking runs entirely on the index, and content is never consulted on the retrieval path. The side-car still gets written, but with content: null it's already just ids + metadata.

One caveat: metadata filtering does read the local table (filter_documents and the embedding_retrieval allowlist at haystack.py:646-649 both walk _u64_to_doc), so metadata must keep living in the side-car — which matches the issue's "metadata stays" scope.

LangChain — the retrieval side is first-class; the ingestion side has no vector-in door

MultiVectorRetriever(vectorstore=..., docstore=..., id_key=...) resolving hits through docstore.mget(ids) is stock LangChain and works with our store today. The gap is exactly what the issue says: the only write paths are add_texts/add_documents (langchain.py:258, :323), which embed exactly the text they store. There is no way to hand the store a precomputed embedding, so a caller who wants text-free storage has two bad options — pass real text (it gets stored in _docs and dumped to the side-car, recreating the bloat) or pass page_content="" (every document embeds identically, search goes semantically dead; nothing raises). An add_embeddings(embeddings, ids, metadatas) method is a small, purely additive change — _store_texts_and_vectors (langchain.py:357) already takes vectors directly, so it's mostly a new public entry point storing "" (or a sentinel) as text. Dict/callable filters walk self._docs (langchain.py:592-601), so metadata again stays local.

LlamaIndex — the framework supports it first-class; our wrapper doesn't honor it yet

This is the interesting one. The framework's own pattern is exactly what you want: a store that declares stores_text=False makes VectorStoreIndex keep nodes in StorageContext.docstore and resolve query-result ids back to full nodes there — SimpleVectorStore itself works this way. Our class even accepts stores_text=False as a constructor kwarg (it's a pydantic field, llama_index.py:213). But the flag is currently cosmetic:

• add() unconditionally stores the full node payload including text — node_to_metadata_dict(node, remove_text=False, ...) at llama_index.py:348-350.
• query() unconditionally reconstructs nodes from that copy via metadata_dict_to_node in _reconstruct_node (llama_index.py:548, called from :829/:852 — the issue's :544/:848 refs have drifted a few lines).
• persist() writes the whole _nodes map to *.nodes.json (llama_index.py:961-980), text included, even when the caller passed stores_text=False.

So the composition functions today (retrieval returns full text — from our copy), but the duplication in memory and on disk remains. Making stores_text=False real means: skip the node_dict payload in add(), return VectorStoreQueryResult(nodes=None, ids=..., similarities=...) from query(), and drop node payloads from persist(). The accepted costs are the same ones SimpleVectorStore imposes: VectorStoreIndex.from_vector_store(vs) raises, and get_nodes() can no longer return populated nodes in that mode. Moderate, well-scoped change.

Agno — the only one with no framework-side mechanism

Agno's VectorDb protocol has no "resolve text elsewhere" hook; search() builds Documents straight from the side-car payload (agno.py:785-814). Two concrete blockers for a text-free setup:

1. The dedup bug is real. _dedup_by_content (agno.py:767-783) keys on md5(doc.content). With blank content, every hit hashes identically and search(limit=3) silently returns 1 result. (With content=None it would AttributeError on .encode().) Deduping on doc.id fixes it and is worth doing independently of this issue.
2. Derived ids make external keying awkward. _derive_doc_id (agno.py:349-353) turns a caller-supplied id into md5(f"{base_id}_{content_hash}"), so an external text store can't key on the id the user passed in — it has to key on doc.name or metadata. That's a documentation item, not a code change (the derivation matches LanceDb's contract deliberately).

One correction to the issue body: the claim that content_hash_exists() state has no durable home isn't quite right under this proposal. _content_hashes is rebuilt from the side-car payloads on load (agno.py:1169-1174), and content_hash is metadata-sized — so as long as the shrunk ids+metadata side-car keeps the content_hash field (it should), the skip-if-already-ingested check survives restarts with path= set. It only breaks if turbovec keeps no state at all, which is beyond what this issue proposes.

Cross-cutting: what must stay in the side-car

Metadata genuinely cannot leave. All four wrappers resolve filters by walking the local table before the kernel search: LangChain _search_vector, LlamaIndex _resolve_allowed_handles/_filters_match, Haystack filter_documents/embedding_retrieval, Agno _meta_matches. The kernel only understands handle allowlists, so metadata is index-adjacent by construction. The shrunk side-car is ids + handles + metadata (+ Agno's content_hash/name/content_id bookkeeping fields) — which is exactly the issue's proposed end state, and removes the text that makes #425's file 5.5x the index.

Bottom line

The "build your own" vision holds: Haystack needs a docs page, LangChain needs one small additive method, LlamaIndex needs its existing stores_text=False flag wired through three methods, and Agno needs a one-line dedup fix plus an honest "subclass and override search" recipe. None of it is a major version — only callers who rely on turbovec handing back text it stored would notice, and for LlamaIndex only when they opt into stores_text=False. Happy to implement any of these in follow-up if you want to sequence it — the LangChain add_embeddings and the Agno dedup fix are independent and low-risk, the LlamaIndex change is the substantive one.

---