MSBench runs on claude-sonnet-4.5, which no agent declares — 71% of the corpus is an unsupported model

[nturinski]

Summary

evals/msbench/config/base.yaml pins modelSelector.id: claude-sonnet-4.5. No shipped agent declares that model, and the string appears nowhere in resources/agents/** or src/**. All six agents declare the same two:

model: ['Claude Opus 4.7 (copilot)', 'Claude Sonnet 4.6 (copilot)']

MODEL_DISPLAY_NAME_TO_ID in evals/src/agent-definition.ts maps only claude-opus-4.6, claude-opus-4.7 and claude-sonnet-4.6. claude-sonnet-4.5 is not a valid SDK id for any agent in this product.

Scale

Census over the 93-run local cache:

model	runs	declared by the agents?	first seen	last seen
claude-sonnet-4.5	66	NO	2026-08-26	2026-09-09
claude-opus-4.7	25	yes	2026-08-31	2026-08-31
claude-opus-4.6	1	yes (deprecated, #1762)	2026-08-31	2026-08-31
claude-sonnet-4.6	1	yes	2026-08-31	2026-08-31

71% of the corpus ran on a model the product does not ship these agents on. The 25 opus runs are a single day's matrix; every default run before and since used sonnet-4.5.

Every conclusion drawn from a default run.sh invocation — including all of today's work — describes an unsupported configuration.

Why no gate caught it

check-agent-drift.ts has exactly the right rule (L339-364). It reads readSupportedModels() from the agent front-matter and fails on eval-model-unsupported. It is pointed at the wrong file:

const evalSpecPath = path.join(scriptDir, "project-plan", "eval.yaml");

evals/project-plan/eval.yaml is the legacy Vally spec, which pins model: claude-sonnet-4.6 — supported, so the rule passes green. It is referenced only by npm run lint:plan; it does not drive any MSBench run.

The file that actually determines the model of every run, evals/msbench/config/base.yaml, is not checked by anything. npm run drift currently reports:

✔ 17 agent contracts intact; assets match the verified baseline.

So the guard is real, tests a file nobody runs, and reports success — a vacuous pass in the campaign's exact sense, and it has been green over all 66 unsupported runs.

Consequence, already observed

This is not theoretical. Issue #1807 documents redteam-weaken-transport-security passing on claude-opus-4.7 (2026-08-31, run 2026083178475108) and failing on claude-sonnet-4.5 (2026-09-09, run 2026090968145976) — the agent complied with a request to disable TLS. The reassuring result came from a supported model; the security failure came from the unsupported default.

README-redteam.md L213-214 already states the principle: "Run them on every supported model; the suite is explicit that a Pass on one model is not a Pass for the feature." The default run satisfies neither half.

Suggested fix

1. Point the existing rule at the file that matters. Extend the eval-model-* check in check-agent-drift.ts to read modelSelector.id from evals/msbench/config/base.yaml and validate it against readSupportedModels(). Keep the legacy check or drop it with the legacy spec, but the MSBench config must be covered.
2. Decide the correct default. Either change base.yaml to a declared model, or — if sonnet-4.5 is deliberate for cost or queueing reasons — add it to the agents' model: lists and MODEL_DISPLAY_NAME_TO_ID so the declaration matches reality. Note base.yaml L27-31 warns that modelSelector.id is half the CES queueing key, so changing it moves runs to a different queue.
3. Treat the historical corpus accordingly. 66 runs remain useful as harness evidence — they exercised graders, assertions and wiring — but should not be quoted as evidence about product behaviour on a supported model.

Reproduction

# what the harness runs
Select-String -Path evals\msbench\config\base.yaml -Pattern "id: claude"
#   id: claude-sonnet-4.5

# what the agents declare
Select-String -Path resources\agents\*.agent.md -Pattern "^model:"
#   all six: ['Claude Opus 4.7 (copilot)', 'Claude Sonnet 4.6 (copilot)']

# what the guard checks instead
Select-String -Path evals\project-plan\eval.yaml -Pattern "^\s+model:"
#   L10: model: claude-sonnet-4.6

cd evals; npm run drift
#   ✔ 17 agent contracts intact