Add CI benchmark suite: latency + recall regression tracking

.github/workflows/benchmark.yml

@@ -0,0 +1,99 @@
+name: Benchmark
+
+permissions:
+  contents: read
+
+# All benchmark runs share one Moss project and the deterministic
+# benchmark-ci-<hash> index, so concurrent jobs could race on first index
+# creation and contaminate each other's latency numbers with cross-job
+# load. Serialize globally; don't cancel a run that is already measuring.
+concurrency:
+  group: moss-benchmark
+  cancel-in-progress: false
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+    inputs:
+      update_baseline:
+        description: 'Update baseline.json with current results'
+        type: boolean
+        default: false
+
+jobs:
+  benchmark:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          # PyPI downloads occasionally break mid-stream on hosted runners
+          # (IncompleteRead / ProtocolError), and pip does not resume a
+          # partial wheel download — retry the whole install a few times.
+          for attempt in 1 2 3; do
+            if pip install -r benchmarks/ci/requirements.txt; then
+              exit 0
+            fi
+            echo "pip install failed (attempt ${attempt}/3) — retrying in 15s"
+            sleep 15
+          done
+          echo "pip install failed after 3 attempts"
+          exit 1
+
+      - name: Run benchmark suite
+        env:
+          MOSS_PROJECT_ID: ${{ secrets.MOSS_PROJECT_ID }}
+          MOSS_PROJECT_KEY: ${{ secrets.MOSS_PROJECT_KEY }}
+          # Fork PRs, and same-repo Dependabot PRs (which GitHub also denies
+          # normal secrets), cannot read repository secrets, so missing
+          # credentials are expected there and the suite may skip. On other
+          # trusted runs (push to main, same-repo human PRs, manual
+          # dispatch) missing secrets make the suite FAIL instead of
+          # passing as a green no-op.
+          ALLOW_BENCHMARK_SKIP: ${{ (github.event_name == 'pull_request' && (github.event.pull_request.head.repo.full_name != github.repository || github.actor == 'dependabot[bot]')) && '1' || '0' }}
+        run: |
+          # Baseline-update runs skip the regression comparison: comparing
+          # against the baseline being replaced would fail the run (and skip
+          # the copy step) exactly when an intentional change moved the numbers.
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ] && \
+             [ "${{ github.event.inputs.update_baseline }}" = "true" ]; then
+            pytest benchmarks/ci/ -v \
+              --benchmark-output=benchmark_results.json
+          else
+            pytest benchmarks/ci/ -v \
+              --benchmark-output=benchmark_results.json \
+              --baseline-file=benchmarks/ci/baseline.json \
+              --latency-threshold=0.20 \
+              --recall-threshold=0.05
+          fi
+
+      - name: Upload results artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-results-${{ github.sha }}
+          path: benchmark_results.json
+          retention-days: 90
+        if: always()
+
+      - name: Update baseline (manual trigger only)
+        if: >-
+          github.event_name == 'workflow_dispatch' &&
+          github.event.inputs.update_baseline == 'true'
+        run: |
+          echo "Copying benchmark_results.json → benchmarks/ci/baseline.json"
+          cp benchmark_results.json benchmarks/ci/baseline.json
+          echo "New baseline (runner copy only — NOT committed):"
+          cat benchmarks/ci/baseline.json
+          echo ""
+          echo "To persist: download the benchmark-results-${{ github.sha }} artifact,"
+          echo "copy it to benchmarks/ci/baseline.json, and commit."

.gitignore

@@ -10,7 +10,7 @@ node_modules/
 dist/
 .next/
 coverage/
-.venv/
+.venv*/
 env/
 venv/
 *.log
@@ -27,3 +27,4 @@ skills-lock.json
 
 # Swift / SwiftPM build artifacts
 .build/
+benchmark_results.json

AGENTS.md

@@ -268,3 +268,10 @@ The `.github/workflows/ci.yml` pipeline runs on push to `main` and on PRs:
 - `python-sdk-test` — matrix over Python 3.10–3.14
 - `javascript-lint` — eslint
 - Separate release workflows publish to PyPI / npm on tagged releases
+
+The `.github/workflows/benchmark.yml` runs on push to `main` and on PRs:
+- Runs the CI benchmark suite (`benchmarks/ci/`) measuring p50/p95/p99 latency and recall@k
+- Compares against `benchmarks/ci/baseline.json` and fails on regressions exceeding thresholds
+- Uploads `benchmark_results.json` as a workflow artifact for each run
+- Supports `workflow_dispatch` with `update_baseline` input to refresh the baseline
+

benchmarks/.gitignore

@@ -1,3 +1,5 @@
 .env
 __pycache__/
 embedding_server/__pycache__/
+ci/__pycache__/
+benchmark_results.json

benchmarks/README.md

@@ -77,4 +77,16 @@ EMBEDDING_DIMENSION=768
 - **Queries**: 15 diverse search queries
 - **Warmup**: 3 rounds (excluded from measurements)
 - **Measured**: 50 rounds x 15 queries = 750 measurements per system
-- **top_k**: 5
+- **top_k**: 5
+
+## CI Benchmark Suite
+
+For automated regression testing in CI, see [`benchmarks/ci/`](ci/).
+The workflow runs on every push to `main` and on PRs; benchmark tests skip
+when Moss credentials are unavailable (such as fork PRs). Authenticated runs
+track p50/p95/p99 latency and recall@k per commit and compare against a
+checked-in baseline, failing the build when regressions exceed the
+configured thresholds. The latency guard activates once a baseline measured
+on CI runners replaces the initial placeholder.
+
+See [`benchmarks/ci/README.md`](ci/README.md) for full documentation.

benchmarks/ci/README.md

@@ -0,0 +1,172 @@
+# CI Benchmark Suite — Latency & Recall Regression Guard
+
+Automated benchmark harness that runs on every push/PR to `main` and catches
+performance regressions before they ship.
+
+## What it measures
+
+| Metric | Description |
+|--------|-------------|
+| **P50 / P95 / P99 latency** | End-to-end query latency (embedding + search) in ms |
+| **Mean / Stdev** | Average and standard deviation of latency |
+| **Recall@5** | Fraction of ground-truth top-5 docs returned in top 5 |
+| **Recall@10** | Fraction of ground-truth top-10 docs returned in top 10 |
+
+All measurements use the Moss built-in embedding model (`moss-minilm`) with a
+1,000-document subset of the benchmark corpus for CI speed.
+
+## Quick start
+
+### Prerequisites
+
+Set `MOSS_PROJECT_ID` and `MOSS_PROJECT_KEY` in your environment (or in a
+`.env` file).
+
+### Run locally
+
+```bash
+# Install dependencies
+pip install -r benchmarks/ci/requirements.txt
+
+# Run the full suite
+pytest benchmarks/ci/ -v \
+  --benchmark-output=benchmark_results.json \
+  --baseline-file=benchmarks/ci/baseline.json
+
+# Skip regression checks (no baseline comparison)
+pytest benchmarks/ci/ -v --benchmark-output=benchmark_results.json
+```
+
+### Regenerate ground truth
+
+Run this when the index data or model changes:
+
+```bash
+python benchmarks/ci/generate_ground_truth.py
+```
+
+This queries Moss with `top_k=50` for each benchmark query and writes the
+expected document IDs to `ground_truth.json`. Commit the updated file.
+After a corpus or model change, pass `--recreate` so the index is rebuilt
+from the current corpus before the ground truth is captured.
+
+`--recreate` deletion is guarded: only the derived `benchmark-ci-<hash>`
+index is deleted without confirmation. If `MOSS_INDEX_NAME` overrides the
+name, deletion additionally requires `--force`, and indexes outside the
+`benchmark-ci-*` namespace are never deleted — so a stray environment
+variable pointing at a shared or production index cannot be destroyed.
+
+> **Note**: the ground truth is a *ranking-stability reference* generated by
+> Moss itself at a known-good commit — not an independent relevance judgment.
+> The recall gate detects changes in retrieval behavior; an intentional
+> relevance improvement will trip it and should be accompanied by a
+> regenerated ground truth in the same PR.
+
+## How regression detection works
+
+The harness compares the current run's metrics against `baseline.json`:
+
+- **Latency**: Fails if P95 increases by more than the threshold (default 20%)
+- **Recall**: Fails if Recall@5 **or** Recall@10 drops by more than the
+  threshold (default 5pp)
+
+The latency guard arms itself from the baseline file:
+
+- The checked-in placeholder declares `"latency_guard": "unarmed"` — the
+  latency test skips with a loud message while recall remains guarded, so
+  trusted runs are not red before the first baseline exists.
+- To arm the guard, download the `benchmark-results-<sha>` artifact from a
+  trusted CI run and commit it as `baseline.json`. Artifacts carry no
+  `latency_guard` flag, so committing one arms the guard automatically.
+- A zero p95 **without** the unarmed flag fails the run as a misconfigured
+  baseline — the guard can never be silently inactive by accident.
+
+Thresholds are configurable via CLI flags:
+
+```bash
+# --latency-threshold: max fractional P95 increase (0.15 = 15%)
+# --recall-threshold:  max absolute recall@5 drop (0.03 = 3pp)
+pytest benchmarks/ci/ -v \
+  --latency-threshold=0.15 \
+  --recall-threshold=0.03
+```
+
+## Updating the baseline
+
+After a legitimate performance change (e.g., model upgrade, index config
+change), update the baseline:
+
+1. Trigger the `Benchmark` workflow manually with `update_baseline=true`.
+   This runs the suite **without** the regression comparison (so an
+   intentional change can't fail its own baseline run) and produces a fresh
+   `benchmark-results-<sha>` artifact. It does **not** commit anything —
+   the runner copy is discarded when the job ends.
+2. Download that artifact, copy it to `benchmarks/ci/baseline.json`, and
+   commit. Baselines should always come from CI runners — latency measured
+   on other hardware is not comparable.
+
+## Index naming and staleness protection
+
+The benchmark index name is derived from two content hashes —
+`benchmark-ci-<data_sig>-<build_fp>`:
+
+- **data signature** covers the model id, `DOC_COUNT`, and the exact corpus
+  slice being indexed;
+- **build fingerprint** covers the installed SDK/bindings versions and the
+  Python SDK source tree, i.e. the code that builds the index.
+
+If any of those inputs change, the name changes and the index is rebuilt
+from the current corpus by the current code — so a stale remote index can
+never be silently benchmarked, and a PR that changes `create_index`,
+document serialization, or the index/model build path always exercises that
+path (an embedding change then surfaces as a recall regression instead of
+passing against old embeddings). Set `MOSS_INDEX_NAME` to override the
+derived name (this bypasses the staleness protection for the index itself;
+the compatibility checks below still apply).
+
+Superseded indexes accumulate in the Moss project as signatures change;
+clean them up occasionally with
+`python benchmarks/ci/generate_ground_truth.py --prune` (only
+`benchmark-ci-*` indexes are ever deleted).
+
+Three compatibility checks keep every comparison honest, and each **fails**
+(never skips) on mismatch:
+
+- `ground_truth.json` embeds the corpus/model **signature** — recall is not
+  evaluated against ground truth generated from different inputs.
+- `ground_truth.json` also embeds a **query-set hash** — adding, removing,
+  or editing `QUERIES` without regenerating trips it.
+- `baseline.json`'s config (signature, query-set hash, doc count, rounds,
+  top_k) must match the current run's config — the regression guard refuses
+  to compare against a baseline captured under different benchmark inputs.
+
+Runs are serialized via a GitHub Actions `concurrency` group (without
+cancelling in-progress runs): all jobs share one Moss project, so parallel
+runs could race on index creation and contaminate each other's latency.
+
+## Credentials policy
+
+- **Trusted CI runs** (push to `main`, same-repo PRs, manual dispatch):
+  missing `MOSS_PROJECT_ID` / `MOSS_PROJECT_KEY` **fails** the suite — a
+  misconfigured secret must not turn the workflow into a green no-op.
+- **Fork PRs**: secrets are not available to forks, so the workflow sets
+  `ALLOW_BENCHMARK_SKIP=1` and the suite skips cleanly.
+- **Local runs** (no `CI` env var): missing credentials skip the suite.
+
+## CI integration
+
+The benchmark runs as a GitHub Actions job (`.github/workflows/benchmark.yml`).
+Results are uploaded as artifacts named `benchmark-results-<sha>` and are
+available for download from the Actions tab.
+
+## File overview
+
+| File | Purpose |
+|------|---------|
+| `test_bench_ci_moss.py` | Main test module (latency, recall, regression guard) |
+| `bench_queries.py` | Shared query set + index config (used by tests and generator) |
+| `conftest.py` | Pytest CLI flags |
+| `generate_ground_truth.py` | One-time ground truth generator |
+| `ground_truth.json` | Pre-computed expected results per query |
+| `baseline.json` | Performance baseline for regression checks |
+| `requirements.txt` | Python dependencies |

benchmarks/ci/baseline.json

@@ -0,0 +1,28 @@
+{
+  "commit": "f4732a1",
+  "timestamp": "2026-07-20T05:35:00+00:00",
+  "latency_guard": "unarmed",
+  "_note": "Recall values are measured (hardware-independent) so the recall guard is active. Latency values are zero and 'latency_guard: unarmed' marks this file as the explicit placeholder: the latency regression test skips (loudly) instead of failing every trusted run. To arm the latency guard, run the Benchmark workflow on a trusted ref, download the benchmark-results-<sha> artifact, and commit it as this file \u2014 artifacts carry no latency_guard flag, so committing one arms the guard automatically. A zero p95 WITHOUT the unarmed flag fails the run as a misconfigured baseline. Latency baselines must come from CI runners; numbers from other hardware are not comparable.",
+  "latency_ms": {
+    "p50": 0,
+    "p95": 0,
+    "p99": 0,
+    "mean": 0,
+    "stdev": 0,
+    "count": 0
+  },
+  "recall": {
+    "recall_at_5": 1.0,
+    "recall_at_10": 1.0,
+    "queries_evaluated": 15
+  },
+  "config": {
+    "doc_count": 1000,
+    "query_rounds": 20,
+    "warmup_rounds": 3,
+    "top_k_latency": 5,
+    "signature": "73d5e83176f7",
+    "query_set_hash": "70d932a5a939",
+    "query_count": 15
+  }
+}