fix(grade-codebase): grade liveness, not presence — close the false-A gap

[markhazlett]

Why

Running /grade-codebase on a real repo returned an A despite the repo having no skills and its agent context living in a global .ai/ folder that hadn't been touched in months (referenced from CLAUDE.md, with no feedback loop for learnings). That's a false positive the skill should never produce.

Root cause

Every D1 signal was a presence/format check (test -f, wc -l, grep for section headers, grep for placeholders). A stale-but-well-formatted context doc passes all of them. The rubric named the quality questions ("is AGENTS.md useful vs. a stub", "are flags used vs. installed", "is the canonical example actually canonical") in its §5 judgment column — but the skill never had to test them, and they were capped at 50%. So presence floated all the way to an A.

The same disease ran through the whole rubric: it graded configured, never enforced/current/honest.

Fix — separate "configured" from "enforced/current/honest", and force the skill to probe the latter

Rubric:

• D1 — signals split into presence (necessary, fakeable) vs. liveness probes (required to score above C): verify 3+ doc claims against the code, freshness vs. code churn, feedback-loop mechanism. Grader must follow references into .ai/, docs/agent/, etc. and grade that content.
• §4 — present-but-misleading scores below absent (a stale doc / empty-green suite / disabled gate is D–F, not C). A D1 of D/F caps the overall grade at C (compounding-context rule — bad onboarding poisons every other dimension, so averaging is wrong here).
• §5 — judgment column reframed as required liveness probes spanning D1/D2/D4/D6/D7/D8/D10 (does the suite assert anything; is the gate a required check vs. continue-on-error; is the lockfile in sync; are flags/loggers used vs. installed; is the canonical example actually canonical).
• §6 — new red flags feat(learn): anti-list defense + verify-before-recommend (G3) #15–18: stale/misleading context, gate configured-but-not-enforced, trivial test suite (all skip/todo/no-assert), lockfile drift.

Skill (SKILL.md + new liveness-probes.md):

• New mandatory step 4 — Run liveness probes (the false-A guard). A probed dimension cannot exceed C until its probe runs; a probe you can't run is not verified, never a pass.
• Honesty rule: never award A on a probed dimension without running its probe.
• Body trimmed back under the 700-word ceiling (699), detail pushed into the sub-file per CONVENTIONS.

Template: per-dimension liveness-probe results block so the verification evidence is surfaced, separate from configuration presence.

Calibration fixtures (the §8 follow-up, now built)

Three deliberately-rigged negative-case mini-repos under .claude/skills/grade-codebase/fixtures/, each targeting one probe / red flag, each with an EXPECTED.md stating the grade it must receive:

Fixture	Failure mode	Must grade
stale-context/	CLAUDE.md (+ referenced .ai/) cites paths/commands the code contradicts	D1 = D/F, overall ≤ C (§6 #15)
empty-suite/	vitest suite green but all skip/todo + expect(true), never calls the code under test	D2 ≤ C (§6 #17)
disabled-gate/	CI lint/typecheck jobs neutered by continue-on-error / || true	D4 ≤ C (§6 #16)

Two validation layers:

• bin/test-grade-fixtures (new) mechanically guards that each fixture remains a valid negative — fails loud if an edit neuters one (verified: simulating an enforced gate makes it fail). This is the regression guard for the calibration set itself; it does not run the model.
• A model-graded /grade-codebase full run against each fixture is the dogfood that confirms the skill marks it down.

VERSION 0.14.1 → 0.14.3.

Remaining open work (tracked in rubric §8)

• Wire the model-graded expectation into bin/skill-eval as a trajectory eval, so a future rubric/skill edit that re-opens the false-A fails automatically (currently the model-graded layer is a manual dogfood).
• The freshness probe isn't mechanically exercised by the fixtures (they're plain dirs, not git repos with backdated history).

Process note

This change was run back through /write-skill (the audit that surfaced the bloated body + missing discovery layer in an earlier commit). The skill is tier: flexible, so the rigid-skill requirements (subagent baseline, rationalization table, eval.yaml) don't apply.

Test plan

• bin/test-frontmatter — 32/32 pass
• bin/test-grade-fixtures — 3/3 fixtures valid negatives; fail-path verified
• wc -w SKILL.md = 699 (< 700 hard ceiling)
• All REQUIRED SUB-FILE references resolve to files on disk
• Build the golden-fixture calibration repos (§8 follow-up)
• Dogfood: run /grade-codebase full against each fixture; confirm the grade matches its EXPECTED.md
• Wire fixtures into bin/skill-eval as an automated trajectory eval

https://claude.ai/code/session_011uDsxgUQ6b68bvRsbtSsvX