Query-side LUT is quantized to 7 bits (max_lut=127), costing 1.5-3.5pp recall@10 — a cross-arch equivalence trade, not a rounding detail

[RyanCodrai]

Found by bug-hunt wave 22 (numerical-stability lens). Severity: medium-high — this is the dominant error term in the whole scoring path, 2-4 orders of magnitude larger than every float-arithmetic effect, and it's undocumented as an accuracy trade.

Method (so the numbers are checkable): an independent NumPy reference reimplementing ChaCha8 + fisher_yates + the 2-round permute/sign/WHT rotation, parsing the v6 .tv bytes directly (codebook, blocked codes, per-vector scales, TQ+ arrays), then computing three scores per (query, vector): S_exact (f64 exact decode), S_lut (bit-faithful u8 LUT per search.rs:1093-1187, summed in exact integer + f64), and S_tv (what search() returns). S_lut reproduces S_tv to ≤3e-6 relative everywhere, so the simulation is faithful.

turbovec/src/search.rs:1170 — let max_lut: f32 = 127.0; with a single global scale = max_span / max_lut.

|S_lut − S_exact| relative to the median top-k score (purely query-side — same codes):

dim	2-bit	3-bit	4-bit
128	3.0 %	4.3 %	5.5 %
768	4.0 %	5.2 %	5.7 %
3072	5.5 %	6.0 %	7.5 %
8192	4.6 %	7.2 %	10.4 %
16384	5.4 %	8.2 %	10.0 %

Practical impact: the returned top-10 differs from an exact f64 decode of the very same codes in 70-100% of queries (set overlap 0.93-0.98). Against true float ground truth at 4 bits: recall@10 0.845 vs 0.870 (dim 768), 0.845 vs 0.860 (3072), 0.855 vs 0.890 (16384) — 1.5-3.5pp of recall discarded by the LUT alone. At 2 bits it's in the noise (code quantization dominates).

The comment at search.rs:1157-1169 says x86 could safely carry max_lut ≈ 255 and that 127 is chosen only so ARM- and x86-built indexes round identically. That cross-arch equivalence is costing ~1 bit of query precision and ~3pp of recall — worth surfacing as a deliberate trade rather than a rounding detail. (The loss isn't ARM's vaddq_u8 cap per se; it's x86 being held down to match.)

Second finding — a single global max_span makes the LUT collapse for concentrated rotated queries (search.rs:1172; mechanism CONFIRMED, real-world reachability low). One scalar is shared by all dim sub-tables, so one wide sub-table starves the rest. Constructed queries whose rotated form is concentrated (unit_query + a·R⁻¹e_j; ROTATION_SEED is a public constant at rotation.rs:82), dim 768, 4-bit:

spike a	rot. concentration	span_max/median	eps_lut / score	top-10 overlap
0	0.12	5	0.057	0.97
1	0.70	41	0.46	0.72
3	0.95	124	2.6	0.67
10	0.995	414	3.5	0.68

At the worst point the LUT error is 3.5× the score magnitude and a third of the top-10 is wrong. Natural queries rotate to concentration ~0.12 and never approach this, so exposure is low — but it fails silently with no warning or fallback, and the rotation is a fixed public constant so it's reachable by construction. A per-sub-table scale, or detecting span_max/median > ~50, would guard it.

Everything else in the float pipeline is sound (measured bounds, recorded so nobody re-derives them): kernel f32 accumulation |S_tv − S_lut| maxes at 2.6e-4 relative at dim 16384 with 0/180 queries reordered, and only 1/10 in a deliberately adversarial near-tie set; the Walsh-Hadamard is flat at ~1.5e-7 relative at every block size including weak-block dims, with no compounding from the 2-round structure; the calibration bias is computed in f64 and the cancellation regime is self-limiting (bias/score falls from 0.76 to 3.4e-5 as the data offset grows, since TQ+ absorbs the offset into shift); and f32 scale storage has 208× overflow headroom at MAX_DIM (worst case 1.64e36 vs f32::MAX 3.4e38 — correctly placed, though the margin in value is only ~14×, not the ~1e22 one might assume).

Readability trap worth a comment: the lo/hi sub-table naming in build_query_neon_lut_from_slice (search.rs:1114-1140) is inverted relative to the packing in build_extract_lut (pack.rs:225-243) — dims [g·cpb, g·cpb+cpn) land in the high nibble but are served by the table labelled lo. Consistent end-to-end, so purely cosmetic, but it cost this agent a debugging cycle and #314 flagged the same inversion independently.

🤖 Generated with Claude Code

[github-actions]

Claude encountered an error after 1s —— View job

---

I'll analyze this and get back to you.

[RyanCodrai]

Set aside for a maintainer decision: resolving this changes encoded bytes and/or published accuracy claims, so it is a trade-off call rather than a bug fix. Flagged during an automated issue-fixing pass on 2026-07-30.

[RyanCodrai]

/summary

[tq-bot]

Plain-English summary: When turbovec searches an index, it converts each incoming query into a small lookup table of whole numbers so the comparison against stored vectors can run fast. Right now that table only uses values up to 127 — roughly half the precision it could — and the number crunching here shows this is the single biggest source of inaccuracy in the whole search path, far bigger than any other rounding effect. In practice it means the top-10 results come back slightly wrong for most queries, costing about 1.5 to 3.5 percentage points of accuracy at the 4-bit setting. The limit isn't an oversight: it exists so that indexes built on ARM and x86 machines produce byte-identical results. The issue argues this is a real accuracy-for-portability trade that should be made deliberately and documented, not treated as a minor rounding detail.

The report also flags a rarer edge case: because one shared scaling factor covers the whole table, an unusually "spiky" query can make the table collapse and return badly wrong results with no warning — unlikely with natural data, but constructible, and easy to guard against. It confirms the rest of the floating-point pipeline is sound, and notes a confusingly inverted "lo/hi" naming in the code (also spotted in #314) that's harmless but has tripped people up. The maintainer has set this aside as a judgment call, since fixing it would change the index format or the published accuracy numbers.

🤖 Generated with Claude Code