Found by bug-hunt wave 37 (consolidated recall measurement). Severity: medium — the headline claim is intact where it's stated, but it reads as universal and isn't.
Using the official suite methodology (n=100k, 1000 queries, seed 42, FAISS IndexPQ m=d/4 @2-bit and m=d/2 @4-bit, nbits=8, IP), with the methodology control passing (openai-1536 2-bit R@1 0.903 vs published 0.891; GloVe 2-bit 0.5695 vs 0.5637; GloVe 4-bit 0.8553 vs 0.8498).
The claimed configs reproduce — all wins or ties at R@1: openai-1536 2-bit 0.903 vs 0.867 (+3.6), 4-bit 0.970 vs 0.969; openai-3072 2-bit 0.931 vs 0.906 (+2.5), 4-bit 0.976 vs 0.971; GloVe 2-bit 0.5695 vs 0.5643, 4-bit 0.8553 vs 0.8410.
Configs where turbovec loses to FAISS IndexPQ (reproducible across two seeds):
| config |
turbovec |
FAISS |
delta |
| isotropic Gaussian d=768, 2-bit, R@1 |
0.353 / 0.393 |
0.450 / 0.490 |
−9.7 / −9.7 pp |
| isotropic Gaussian d=1536, 2-bit, R@1 |
0.440 / 0.363 |
0.450 / 0.453 |
−1.0 / −9.0 pp |
| isotropic Gaussian d=768/1536, 4-bit, R@1 |
0.757–0.783 |
0.803–0.817 |
−3.3 to −6.0 pp |
| sparse d=1032 (weak block), 2-bit, R10@10 |
0.546 |
0.757 |
−21.1 pp (turbovec still wins R@1) |
| GloVe d=200, 2-bit, R10@10 |
0.6245 |
0.6325 |
−0.8 pp |
Important context that argues against over-reacting: on realistic anisotropic data turbovec crushes FAISS (d=768 2-bit R@1 0.697 vs 0.120), so the isotropic-Gaussian loss is a synthetic-regime effect, not a user-facing break. The README claim is correct for real embeddings. The gap is that it reads as universal, and "isotropic Gaussian" and "weak-block sparse" are the two counterexamples a skeptical reader would find. A sentence scoping it to real embedding distributions would close this. (Complements #310, which found the bimodal/one-hot losses.)
One methodological note worth recording for anyone re-measuring #332: I tried to isolate the 7-bit query LUT's contribution by injecting score noise at the measured 4–5.7% relative level. That costs 25–45 pp — far more than #332's measured 1.5–3.5 pp. Not a contradiction: it shows the real LUT error is strongly correlated across candidates (one LUT per query), which is exactly why its ranking damage is small. The emulation is invalid as a proxy; #332's number stands as measured there.
Also confirmed and sharpened: #335's magnitude cliff fires a decade earlier than reported — at query scale 1e-10, not 1e-11 (R10@10 0.926 → 0.0003 on the base config at 4-bit).
Recommended recall fix order (see the measurement table posted on #309): weak-block dims first (13–38 pp, up to 50 pp at R@1, and it's a free fix), then the magnitude cliff (#335 — cheap, catastrophic when hit), then the 7-bit LUT (#332 — 1.5–3.5 pp for every 4-bit user, the largest unconditional term after the irreducible quantization floor), and only then the calibration warm-up (#317/#366 — real but 1.5–1.9 pp at 2-bit on real data and ≤0.4 pp at 4-bit; worth fixing for its silence and irreversibility, not its magnitude).
🤖 Generated with Claude Code
Found by bug-hunt wave 37 (consolidated recall measurement). Severity: medium — the headline claim is intact where it's stated, but it reads as universal and isn't.
Using the official suite methodology (n=100k, 1000 queries, seed 42, FAISS
IndexPQm=d/4 @2-bit and m=d/2 @4-bit, nbits=8, IP), with the methodology control passing (openai-1536 2-bit R@1 0.903 vs published 0.891; GloVe 2-bit 0.5695 vs 0.5637; GloVe 4-bit 0.8553 vs 0.8498).The claimed configs reproduce — all wins or ties at R@1: openai-1536 2-bit 0.903 vs 0.867 (+3.6), 4-bit 0.970 vs 0.969; openai-3072 2-bit 0.931 vs 0.906 (+2.5), 4-bit 0.976 vs 0.971; GloVe 2-bit 0.5695 vs 0.5643, 4-bit 0.8553 vs 0.8410.
Configs where turbovec loses to FAISS IndexPQ (reproducible across two seeds):
Important context that argues against over-reacting: on realistic anisotropic data turbovec crushes FAISS (d=768 2-bit R@1 0.697 vs 0.120), so the isotropic-Gaussian loss is a synthetic-regime effect, not a user-facing break. The README claim is correct for real embeddings. The gap is that it reads as universal, and "isotropic Gaussian" and "weak-block sparse" are the two counterexamples a skeptical reader would find. A sentence scoping it to real embedding distributions would close this. (Complements #310, which found the bimodal/one-hot losses.)
One methodological note worth recording for anyone re-measuring #332: I tried to isolate the 7-bit query LUT's contribution by injecting score noise at the measured 4–5.7% relative level. That costs 25–45 pp — far more than #332's measured 1.5–3.5 pp. Not a contradiction: it shows the real LUT error is strongly correlated across candidates (one LUT per query), which is exactly why its ranking damage is small. The emulation is invalid as a proxy; #332's number stands as measured there.
Also confirmed and sharpened: #335's magnitude cliff fires a decade earlier than reported — at query scale 1e-10, not 1e-11 (R10@10 0.926 → 0.0003 on the base config at 4-bit).
Recommended recall fix order (see the measurement table posted on #309): weak-block dims first (13–38 pp, up to 50 pp at R@1, and it's a free fix), then the magnitude cliff (#335 — cheap, catastrophic when hit), then the 7-bit LUT (#332 — 1.5–3.5 pp for every 4-bit user, the largest unconditional term after the irreducible quantization floor), and only then the calibration warm-up (#317/#366 — real but 1.5–1.9 pp at 2-bit on real data and ≤0.4 pp at 4-bit; worth fixing for its silence and irreversibility, not its magnitude).
🤖 Generated with Claude Code