feat(embedding): optionally truncate oversized embeddings to configured dims

[JustinBacher]

Problem

Matryoshka (MRL) embedding models served behind OpenAI-compatible endpoints (llama.cpp, some vLLM builds) ignore the dimensions= request parameter and always return their native dimension. Honcho then hard-fails in _validate_embedding_dimensions on the mismatch, so these models are unusable — even though truncating an MRL vector to a smaller size is a valid, designed-in operation.

Concretely: a model like llama-nemotron-embed-1b-v2 emits 2048 dims. If you configure EMBEDDING_VECTOR_DIMENSIONS=1536, honcho sends dimensions=1536, the server ignores it and returns 2048, and every embedding raises Embedding dimension mismatch.

This also collides with pgvector's 2000-dimension HNSW index cap: native dims above 2000 can't be indexed at all, so truncating to a supported MRL size (e.g. 1024/1536) is the only way to keep an ANN index.

Change

Add an opt-in EMBEDDING_TRUNCATE_TO_DIMENSIONS flag (default false). When enabled and the provider returns a vector larger than EMBEDDING_VECTOR_DIMENSIONS, keep the leading components and L2-renormalize so cosine / inner-product similarity stays well-defined, then validate. When disabled (default), behavior is unchanged — a genuinely misconfigured model still fails loudly.

• src/config.py — new EmbeddingSettings.TRUNCATE_TO_DIMENSIONS field, documented.
• src/embedding_client.py — _l2_truncate() helper + _fit_embedding_dimensions(), applied at both OpenAI call sites (single + batch); threaded through the singleton and its cache signature. Gemini path unchanged (it already honors output_dimensionality).
• .env.template — document the flag.
• tests/llm/test_embedding_client.py — 4 new tests (helper slice+renorm, zero-norm guard, client truncates when enabled, client still rejects when disabled).

Testing

• uv run pytest tests/llm/test_embedding_client.py — 20 passed.
• uv run basedpyright src/embedding_client.py src/config.py — 0 errors.
• uv run ruff check / ruff format — clean.
• Verified end-to-end against a local llama.cpp serving llama-nemotron-embed-1b-v2 (native 2048): with the flag on and EMBEDDING_VECTOR_DIMENSIONS=1536, embeddings come back 1536-dim, unit-norm, and store/search in a vector(1536) pgvector column with an HNSW index.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added an optional setting to truncate oversized embedding vectors to the configured dimensions.
  • Truncated vectors are L2-renormalized for consistent results.
  • The option remains disabled by default, preserving existing dimension-mismatch errors.

• Documentation

  • Added configuration guidance for providers that ignore requested embedding dimensions.

• Tests

  • Added coverage for truncation, normalization, zero vectors, and mismatch handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0c4ae345-d60f-47b9-9410-df0e7b7e0b78

📥 Commits

Reviewing files that changed from the base of the PR and between 055d73b and 02edbe0.

📒 Files selected for processing (4)

• .env.template
• src/config.py
• src/embedding_client.py
• tests/llm/test_embedding_client.py

---

Walkthrough

Adds an opt-in embedding configuration that truncates oversized provider vectors to the configured dimension, L2-renormalizes them, and preserves strict dimension validation when disabled. Single and batch OpenAI paths use the new behavior, with corresponding unit and integration tests.

Changes

Embedding dimension handling

Layer / File(s)	Summary
Dimension truncation configuration .env.template, src/config.py	Documents and adds TRUNCATE_TO_DIMENSIONS, defaulting to False.
Embedding client truncation flow src/embedding_client.py	Adds optional truncation and L2-renormalization for single and batch OpenAI embeddings, and wires the setting into client initialization and reinitialization.
Truncation and mismatch tests tests/llm/test_embedding_client.py	Tests vector truncation, normalization, zero-vector handling, and strict mismatch errors when disabled.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIEmbeddingAPI
  participant _EmbeddingClient
  participant _l2_truncate
  OpenAIEmbeddingAPI->>_EmbeddingClient: return oversized embedding
  _EmbeddingClient->>_l2_truncate: truncate leading components
  _l2_truncate-->>_EmbeddingClient: return L2-renormalized vector
  _EmbeddingClient-->>OpenAIEmbeddingAPI: return fitted embedding

Loading

Suggested reviewers: rajat-ahuja1997

Poem

A rabbit trims vectors, neat and bright,
Renormalizing them just right.
When truncation’s off, errors speak,
When it’s on, dimensions fit sleek.
Hop, hop—tests guard the tweak!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: optional truncation of oversized embeddings to configured dimensions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[karaokedjodua]

Finn-loop review of 02edbe0

CI: not configured (CodeRabbit review passed; no required checks on this branch)
Mergeability: BLOCKED — stale branch protection (mergeable=MERGEABLE but mergeStateStatus=BLOCKED; gh pr checks --required reports "no required checks reported")

Review

Adds an opt-in EMBEDDING_TRUNCATE_TO_DIMENSIONS flag (default false) that truncates oversized MRL-trained embeddings to the configured dimension and L2-renormalizes. Four files touched: config, embedding client (single + batch paths), env template, and tests. Code is clean, focused, and well-tested.

1. Must fix before merge

[CI] No required CI configured on the feat/embedding-mrl-truncation branch. gh pr checks --required reports "no required checks reported" and mergeStateStatus is BLOCKED while mergeable is MERGEABLE — this is the stale branch-protection rule pattern. An admin must update or remove the stale rule before this can merge.

2. Should fix soon

None.

3. Safe to merge

Code is clean — the change is minimal, correctly scoped, uses a safe default (off), has coverage for the truncation path, the zero-norm edge case, and the backward-compatible rejection path. The CI infrastructure gap is the only blocker.

Labels needed: needs-human-review, loop-changes-requested