@@ -21,13 +21,32 @@ Machine layers are DONE; the items below need Alan.
2121 "baseline" was a one-line question — mode-aware baselines fix this going
2222 forward; treat legacy compose runs as unscoreable for scope discipline.
2323
24+ ## Judge batch — DONE (2026-06-10 evening, live claude CLI)
25+
26+ CLI authenticated via ` claude setup-token ` (1-year token in the macOS
27+ keychain, exported by ~ /.zshrc). Judged the 7 gate-passing technical runs
28+ (the 2 personally-named passing runs were left for Alan):
29+
30+ | Verdict | Count |
31+ | ---------| -------|
32+ | improved | 6 |
33+ | position_biased (no claim) | 1 |
34+ | regressed / tie / invalid-evidence | 0 |
35+
36+ All evidence quotes verified verbatim. First batch surfaced a verifier bug
37+ (trailing-space quote rejection -> 6/7 false "invalid-evidence"); fixed in
38+ commit 4adbd9f + b2ad043 era — see judge-bounce.sh history.
39+
40+ ` improved ` rate on judged runs: ** 6/7 (86%)** — clears the >=60% half of
41+ the v1.4 seed trigger. The judge↔human agreement half remains below.
42+
2443## Blocked on a human (in order)
2544
26- - [ ] ** Log in the claude CLI on the Mac** (` claude ` , then ` /login ` ). The
27- judge batch errored cleanly: "Not logged in". Then:
28- ` bash evals/calibrate-bounce.sh --runs-dir runs/ --judge --max-judge 10 `
2945- [ ] ** Blind-sample 5 judged runs** using the worksheet at the bottom of the
3046 calibration report (verdicts hidden until your call is recorded).
47+ Candidates: the 6 "improved" runs + optionally re-judge the 2
48+ personally-named passing runs first
49+ (` bash evals/judge-bounce.sh --run-dir runs/<name> ` ).
3150- [ ] ** Compare judge↔human agreement.** ≥4/5: trust the judge for routine
3251 gating. ≤2/5 or systematic self-preference: wire a third-family judge
3352 (OpenRouter) before relying on verdicts.
0 commit comments