Skip to content

Commit 0e949f4

Browse files
Alan Shurafaclaude
authored andcommitted
docs: record live judge batch results — 6/7 improved, 0 regressed (calibration tracker)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b2ad043 commit 0e949f4

1 file changed

Lines changed: 22 additions & 3 deletions

File tree

.planning/VERIFY-BOUNCE-CALIBRATION.md

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,32 @@ Machine layers are DONE; the items below need Alan.
2121
"baseline" was a one-line question — mode-aware baselines fix this going
2222
forward; treat legacy compose runs as unscoreable for scope discipline.
2323

24+
## Judge batch — DONE (2026-06-10 evening, live claude CLI)
25+
26+
CLI authenticated via `claude setup-token` (1-year token in the macOS
27+
keychain, exported by ~/.zshrc). Judged the 7 gate-passing technical runs
28+
(the 2 personally-named passing runs were left for Alan):
29+
30+
| Verdict | Count |
31+
|---------|-------|
32+
| improved | 6 |
33+
| position_biased (no claim) | 1 |
34+
| regressed / tie / invalid-evidence | 0 |
35+
36+
All evidence quotes verified verbatim. First batch surfaced a verifier bug
37+
(trailing-space quote rejection -> 6/7 false "invalid-evidence"); fixed in
38+
commit 4adbd9f + b2ad043 era — see judge-bounce.sh history.
39+
40+
`improved` rate on judged runs: **6/7 (86%)** — clears the >=60% half of
41+
the v1.4 seed trigger. The judge↔human agreement half remains below.
42+
2443
## Blocked on a human (in order)
2544

26-
- [ ] **Log in the claude CLI on the Mac** (`claude`, then `/login`). The
27-
judge batch errored cleanly: "Not logged in". Then:
28-
`bash evals/calibrate-bounce.sh --runs-dir runs/ --judge --max-judge 10`
2945
- [ ] **Blind-sample 5 judged runs** using the worksheet at the bottom of the
3046
calibration report (verdicts hidden until your call is recorded).
47+
Candidates: the 6 "improved" runs + optionally re-judge the 2
48+
personally-named passing runs first
49+
(`bash evals/judge-bounce.sh --run-dir runs/<name>`).
3150
- [ ] **Compare judge↔human agreement.** ≥4/5: trust the judge for routine
3251
gating. ≤2/5 or systematic self-preference: wire a third-family judge
3352
(OpenRouter) before relying on verdicts.

0 commit comments

Comments
 (0)