Feature/embedding result cache

.dockerignore

@@ -0,0 +1,13 @@
+# Keep the build context small: the venv and result dirs are not needed to run.
+.venv/
+**/__pycache__/
+*.pyc
+.git/
+.pytest_cache/
+docs/
+# Benchmark output dirs (regenerated by the runs themselves). Comment a line
+# back in if you want the reference JSONs baked into the image for comparison.
+bench_*/
+!bench_*/aggregate.py
+!bench_*/paired.py
+!bench_*/paired_stats.py

.gitignore

@@ -138,4 +138,11 @@ dmypy.json
 **/example.db
 **/.chroma
 docs/references/*
-!docs/references/index.rst
+!docs/references/index.rst
+
+CLAUDE.md
+.claude/*
+.env
+
+docs
+onnx_dynamic

BENCHMARKING.md

@@ -0,0 +1,195 @@
+# How to benchmark
+
+Reproduction guide for the four contributions in *Cost-Aware W-TinyLFU for Semantic LLM Caches*
+(`docs/report.tex`). Every command below is deterministic given a `--seed`; the paper's headline
+numbers come from replaying seeds 0–6 and pairing per-seed.
+
+| # | Contribution | Script | Result dir |
+|---|---|---|---|
+| 1 | `CA_W_TINYLFU` cost-aware eviction | `examples/benchmark/benchmark_lmsys.py` | `bench_lmsys/`, `bench_wildchat/` |
+| 2 | SBERTMRL + HNSW/SQ8 storage | `examples/benchmark/benchmark_qqp.py` | `bench_real_100k/` |
+| 3 | `EmbeddingDispatcher` fan-out | `examples/benchmark/benchmark_embedding_dispatcher.py` | `bench_embedding_dispatcher/` |
+| 6 | `CachedEmbedding` exact-repeat cache | `examples/benchmark/benchmark_embedding_cache.py` | `bench_embedding_cache/` |
+
+The reference JSON logs for every table in the report are already checked in under those `bench_*/`
+directories — a grader can inspect the sample outputs without re-running anything.
+
+---
+
+## Setup
+
+### Option A — Docker (recommended for reproducibility)
+
+```bash
+docker build -t gptcache-bench .
+# writes results into ./out on the host
+docker run --rm -v "$PWD/out:/app/out" \
+  gptcache-bench examples/benchmark/benchmark_lmsys.py --dataset ultrachat --out out/results.json
+```
+
+`ultrachat` is public and needs no credentials. The gated datasets (`lmsys`, `wildchat`) need a
+HuggingFace token:
+
+```bash
+docker run --rm -e HF_TOKEN=hf_xxxxx -v "$PWD/.hf_cache:/app/.hf_cache" -v "$PWD/out:/app/out" \
+  gptcache-bench examples/benchmark/benchmark_lmsys.py --dataset lmsys --out out/lmsys.json
+```
+
+Mount `.hf_cache` so datasets/models download once and persist across runs.
+
+### Option B — local virtualenv
+
+```bash
+python -m venv .venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+pip install -r examples/benchmark/requirements.txt
+pip install faiss-cpu onnxruntime psutil
+pip install -e .
+```
+
+Requires Python ≥ 3.8.1. For the gated datasets: `huggingface-cli login` (or `export HF_TOKEN=...`).
+
+---
+
+## Contribution 1 — cost-aware eviction (`CA_W_TINYLFU`)
+
+Full-stack replay (SBERT encode → FAISS search → similarity gate → on-miss save with eviction) against
+a fresh disk-backed SQLite+FAISS store. Primary metric: **cost-weighted hit rate**.
+
+**Stationary regime** (fixed Zipf, frequency's regime — Table 2 / `tab:stationary`):
+
+```bash
+# flat skew z11 (drift-zipf 1.1), sharp skew z15 (drift-zipf 1.5); sweep seeds 0..6
+for s in 0 1 2 3 4 5 6; do
+  python examples/benchmark/benchmark_lmsys.py --dataset lmsys \
+    --drift --drift-rotate 0 --drift-zipf 1.1 \
+    --cache-sizes 25,50,100 --policies LRU,GDSF,WTINYLFU_FREQ,CA_W_TINYLFU \
+    --repeats 1 --seed $s --out bench_lmsys/win_z11_seed$s.json
+done
+```
+
+Swap `--dataset wildchat` for the external-validity replication (`bench_wildchat/`).
+
+**Drift regime** (rotating hot set — Table 1 / `tab:drift`). Decay is the lever: `--virtual-clock-sec 0`
+is decay-off (ADAPT loses), `120` is decay-on (ADAPT wins):
+
+```bash
+# decay-on, gentle drift, adaptive window, seeds 0..6
+for s in 0 1 2 3 4 5 6; do
+  python examples/benchmark/benchmark_lmsys.py --dataset lmsys \
+    --drift --drift-rotate 3000 --drift-shift 0.02 --drift-zipf 1.2 \
+    --virtual-clock-sec 120 --adaptive-window \
+    --cache-sizes 100 --policies LRU,CA_W_TINYLFU --repeats 1 --seed $s \
+    --out bench_lmsys/decaytuned_sh02_vc120_seed$s.json
+done
+```
+
+**GDSF head-to-head** (Table 3 / `tab:gdsf`): include `GDSF` in `--policies` at `--drift-zipf 1.1`
+(stationary flat) and `1.5` (sharp). **Cost-priority dial** (Section 6.6): add
+`--cost-priority-sweep 0,0.25,0.5,0.75,1 --cache-sizes 200`.
+
+**Aggregation** (paired stats → the deltas and Wilcoxon p-values in the report):
+
+```bash
+python bench_lmsys/paired.py       # paired mean delta, sign count, exact Wilcoxon, 95% CI
+python bench_lmsys/aggregate.py    # per-cell summary tables
+```
+
+---
+
+## Contribution 2 — storage (SBERTMRL + HNSW/SQ8)
+
+100K-vector real-encoder run over QQP, 2000 true-positive + 2000 false-positive probes per cell.
+Cells map to Table 5 (`tab:storage`): A = ONNX/768/Flat baseline, E = MRL/256/HNSW+PQ, G =
+MRL/256/HNSW+SQ8, J = static-encoder variant.
+
+```bash
+python examples/benchmark/benchmark_qqp.py --scale 100000 --cells A,E,G,J \
+  --threshold-sweep 0.86,0.88,0.90,0.92,0.94 --out bench_real_100k/results.json
+```
+
+The `--threshold-sweep` is what shows the false-positive gap is mostly a threshold artifact
+(report Section 8, "Accuracy is mostly a threshold artifact"). Reference logs: `bench_real_100k/cell_*/`.
+
+---
+
+## Contribution 6 — exact-repeat embedding cache (`CachedEmbedding`)
+
+Per-query replay (one `to_embeddings()` call at a time) timing a plain SBERT encoder against the same
+encoder wrapped in `CachedEmbedding`. Reports the **measured** exact-duplicate ratio, hit rate, and
+speedup — Table 6 (`tab:embcache`).
+
+```bash
+# structural smoke test — no network, no model download
+python examples/benchmark/benchmark_embedding_cache.py --dataset synthetic
+
+# paper run: real SBERT, UltraChat replay, seeds 0..6
+for s in 0 1 2 3 4 5 6; do
+  python examples/benchmark/benchmark_embedding_cache.py \
+    --dataset ultrachat --n-queries 5000 --cache-size 10000 --seed $s \
+    --out bench_embedding_cache/results_seed$s.json
+done
+python bench_embedding_cache/paired_stats.py   # 7/7 seeds, geo-mean 3.80x, Wilcoxon p=0.016
+```
+
+**Negative control** (the honesty check): a single pass over distinct prompts has a 0% duplicate ratio
+and correctly yields ~1.02× — see `bench_embedding_cache/README.md`.
+
+---
+
+## Contribution 3 — dispatcher (`EmbeddingDispatcher`)
+
+Embedding *throughput* under concurrent load: a single-process encoder (concurrent callers contend on
+one shared model) vs. a dispatcher that fans across worker processes. Reports the crossover in Table 7 —
+the dispatcher loses below ~15 concurrent callers (IPC overhead), crosses over in an unstable ~15–30
+region, and wins above it, up to ~3.4× at 100, at a fixed ~5 GB RSS cost. The crossover point is
+host-dependent; numbers are from one machine.
+
+```bash
+# offline plumbing check -- real worker processes, synthetic (no-download) encoder
+python examples/benchmark/benchmark_embedding_dispatcher.py --dataset synthetic \
+  --n-prompts 40 --concurrency-levels 1,4
+
+# paper run: 3 runs of the 8-level crossover sweep (200 prompts, 8 workers)
+for r in 1 2 3; do
+  python examples/benchmark/benchmark_embedding_dispatcher.py --dataset ultrachat \
+    --n-prompts 200 --concurrency-levels 1,10,20,30,40,50,75,100 \
+    --out bench_embedding_dispatcher/results_fine_run$r.json
+done
+
+# aggregate the 3 runs into the per-concurrency mean +/- std of Table 7
+python bench_embedding_dispatcher/finegrained_stats.py "bench_embedding_dispatcher/results_fine_run*.json"
+```
+
+Memory numbers (`rss_mb`, summed over the worker children) require `psutil` (`pip install psutil`);
+without it the run still works and reports `0.0`. The dispatcher itself
+(`gptcache/embedding/dispatcher.py`) is unit-tested (`tests/unit_tests/embedding/test_dispatcher.py`)
+and usable directly:
+
+```python
+from gptcache.embedding.dispatcher import EmbeddingDispatcher
+
+def make_sbert():                       # must be a picklable module-level fn (Windows spawn)
+    from gptcache.embedding import SBERT
+    return SBERT("all-MiniLM-L6-v2")
+
+with EmbeddingDispatcher(make_sbert, num_workers=8) as d:
+    vecs = d.to_embeddings_batch(list_of_texts)
+```
+
+---
+
+## Metrics recorded
+
+Each run emits JSON with, per configuration: `cost_weighted_hit_rate` (primary), `hit_rate`,
+`token_saving_ratio`, latency `p50/p95/p99`, throughput (qps), and peak memory (`--track-mem`, or RSS
+for the dispatcher). The paired helper scripts turn per-seed JSON into the report's deltas: mean paired
+delta, sign-consistency count, exact two-sided Wilcoxon p, and a paired-t 95% CI.
+
+## Running the unit tests
+
+```bash
+pytest tests/unit_tests/manager/eviction/test_ca_w_tinylfu.py \
+       tests/unit_tests/embedding/test_cached_embedding.py \
+       tests/unit_tests/embedding/test_dispatcher.py
+```

Dockerfile

@@ -0,0 +1,38 @@
+# Reproducible benchmark environment for the Cost-Aware W-TinyLFU project.
+# Build:  docker build -t gptcache-bench .
+# Run:    docker run --rm -v "$PWD/out:/app/out" gptcache-bench <benchmark args>
+# See BENCHMARKING.md for the exact commands that regenerate each paper result.
+
+FROM python:3.11-slim
+
+# libgomp1 is the one system lib the faiss-cpu wheel links against at runtime.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends libgomp1 \
+ && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+# Install deps first so the layer caches across code edits.
+# - requirements.txt            : gptcache core (numpy, cachetools, requests)
+# - examples/benchmark/*.txt    : tiktoken, datasets, sentence-transformers (pulls torch)
+# - faiss-cpu / onnxruntime      : default vector index + default ONNX encoder (storage cell A)
+COPY requirements.txt ./requirements.txt
+COPY examples/benchmark/requirements.txt ./bench-requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt \
+ && pip install --no-cache-dir -r bench-requirements.txt \
+ && pip install --no-cache-dir faiss-cpu onnxruntime psutil
+
+COPY . /app
+RUN pip install --no-cache-dir -e .
+
+# HuggingFace dataset/model cache. Mount a volume here to avoid re-downloading:
+#   docker run -v "$PWD/.hf_cache:/app/.hf_cache" ...
+ENV HF_HOME=/app/.hf_cache
+
+# ultrachat is public and needs no token. LMSYS-Chat-1M and WildChat-1M are
+# gated: pass a token with  docker run -e HF_TOKEN=hf_xxxxx ...
+# (the datasets library reads HF_TOKEN automatically).
+
+# Default: print the eviction benchmark's help so a bare `docker run` is self-documenting.
+ENTRYPOINT ["python"]
+CMD ["examples/benchmark/benchmark_lmsys.py", "--help"]

README_SUBMISSION.md

@@ -0,0 +1,77 @@
+# Cost-Aware W-TinyLFU for Semantic LLM Caches — submission
+
+Eitan Dubinski, Ayala Egoz.
+
+This repository is a fork of [GPTCache](https://github.com/zilliztech/GPTCache) with four
+contributions to its eviction, storage, and embedding layers. The full write-up is
+**`docs/report.tex`**; this README is the entry point for building, testing, and reproducing it.
+
+## Contributions at a glance
+
+| # | Contribution | Code | Report section | Unit test |
+|---|---|---|---|---|
+| 1 | `CA_W_TINYLFU` — cost-aware W-TinyLFU eviction | `gptcache/manager/eviction/ca_w_tinylfu.py` | *Design* / *Results* | `tests/unit_tests/manager/eviction/test_ca_w_tinylfu.py`, `..._routing.py` |
+| 2 | `SBERTMRL` + FAISS HNSW/SQ8 storage compression | `gptcache/embedding/sbert_mrl.py`, `gptcache/manager/vector_data/faiss.py` | *Storage co-contribution* | `tests/unit_tests/manager/test_local_index.py` |
+| 3 | `EmbeddingDispatcher` — multiprocess encode fan-out | `gptcache/embedding/dispatcher.py` | *Embedding-layer efficiency* | `tests/unit_tests/embedding/test_dispatcher.py` |
+| 6 | `CachedEmbedding` — exact-repeat encode cache | `gptcache/embedding/cached_embedding.py` | *Embedding-layer efficiency* | `tests/unit_tests/embedding/test_cached_embedding.py` |
+
+(Numbering follows the project directions; #4/#5 were not pursued.)
+
+## Install
+
+Requires Python ≥ 3.8.1.
+
+```bash
+python -m venv .venv && . .venv/bin/activate          # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+pip install -r examples/benchmark/requirements.txt
+pip install faiss-cpu onnxruntime psutil
+pip install -e .
+```
+
+A Docker path (recommended for a clean reproduction) is documented in
+[`BENCHMARKING.md`](BENCHMARKING.md#setup). The gated datasets (`lmsys`, `wildchat`) need a
+HuggingFace token (`huggingface-cli login` or `export HF_TOKEN=...`); `ultrachat` and `qqp` are public.
+
+## Verify the build (no downloads)
+
+```bash
+# unit tests for all four contributions
+pytest tests/unit_tests/manager/eviction/test_ca_w_tinylfu.py \
+       tests/unit_tests/manager/eviction/test_ca_w_tinylfu_routing.py \
+       tests/unit_tests/manager/test_local_index.py \
+       tests/unit_tests/embedding/test_cached_embedding.py \
+       tests/unit_tests/embedding/test_dispatcher.py
+
+# offline benchmark smoke tests (real code paths, synthetic no-download encoders)
+python examples/benchmark/benchmark_embedding_cache.py --dataset synthetic
+python examples/benchmark/benchmark_embedding_dispatcher.py --dataset synthetic --n-prompts 40 --concurrency-levels 1,4
+```
+
+## Reproduce the paper
+
+The reference JSON logs behind every table are already checked in under the `bench_*/` directories,
+so a grader can inspect the numbers without re-running anything. To regenerate them:
+
+| # | Benchmark script | Results dir |
+|---|---|---|
+| 1 | `examples/benchmark/benchmark_lmsys.py` | `bench_lmsys/`, `bench_wildchat/` |
+| 2 | `examples/benchmark/benchmark_qqp.py` | `bench_real_100k/` |
+| 3 | `examples/benchmark/benchmark_embedding_dispatcher.py` | `bench_embedding_dispatcher/` |
+| 6 | `examples/benchmark/benchmark_embedding_cache.py` | `bench_embedding_cache/` |
+
+**Full per-contribution commands, seeds, and aggregation scripts are in
+[`BENCHMARKING.md`](BENCHMARKING.md).** Headline numbers come from replaying seeds 0–6 and pairing
+per-seed (mean paired delta, sign count, exact Wilcoxon *p*, 95% CI).
+
+> The dispatcher throughput crossover (#3) is host-dependent — the speedup magnitude and break-even
+> point vary with core count and machine load. Reported numbers are from a single machine; the ~5 GB
+> RSS floor is the stable, reproducible cost.
+
+## Building the report
+
+```bash
+cd docs && pdflatex report.tex && pdflatex report.tex   # twice, for refs
+```
+
+`docs/report.tex` is self-contained (figures under `docs/figures/`, bibliography inline).