feat(bench): verification-stress suite for the EBM mechanism experiment - #7953
Merged
Conversation
A local dashboard for watching a bench run while it writes trajectories: 'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory per poll behind /api/state and serves one self-contained dark instrument page. The run renders as a strip of task segments; each task as a recessed per-round trace whose win/break polarity is carried by geometry (up/down off the baseline) because the green/red pair alone fails CVD validation. The renderer is incremental - DOM nodes are keyed, new rounds pop in, numbers tween, hover survives polling. A replay engine scrubs the whole run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a finished run can be watched back round by round. Suite awareness lists queued tasks as ghosts and prices the remaining time. summarizeTrajectory splits into scanTrajectoryFile + finish so the dashboard can read the raw outcome series the summary discards; rounds gain their observation timestamp for the replay timeline.
The p90 tail decomposed into pure model-generation time, but the digest could not say what those seconds bought. Executor reasoning/completion tokens now join each model round: every classified round carries a roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions), totals accumulate per run, and rounds whose gap crosses 8s enter a slow-round census. Delegation rounds get their own classification - previously they read as mutations, hiding the cost story of a subagent call inside a bugfix. The report gains one Cognition line: totals, reasoning/solved, the output rate p50/p90 (uniform rates indict token volume, not serving), the slow-round share of model time, and delegation cost. Backfills over already-recorded trajectories - usage events have carried reasoning and completion tokens all along. Interval math moves to intervals.go to keep trajectory.go under the file-size ceiling.
One research call cost 65% of the worst benchmark task's wall clock - a config-format bugfix delegating to an external-research subagent. The boundary: local mutation-shaped turns get no expensive research unless the user asked for it, the call cites an external source, or the turn is not a local fix at all. Shadow means observed, never enforced: each gated call records a content-free verdict (tool, verdict, reason, turn intent) through a new sink capability into the trajectory. The turn text judged is the bounded classifier source the delivery gates already store, so the shadow and the gates read the same words. e2ebench joins verdicts with delegation tool durations (order- independent - the verdict lands after the result) and reports one line: gated calls, would-deny rate, and the subagent time behind denied calls. A/B enforcement waits for this shadow data.
The failed benchmark task spent 75s thinking and 0.5s in tools - edits without ever asking the environment whether the hypothesis held. The shadow scorer now tracks that pattern: a mutation opens verification debt, every round without a discriminating observation ages it, and a discriminating observation settles it. Discriminating is deliberately broader than delivery verification - a verification-classified command, or any command exercising a file a mutation touched this turn (the repro-script case). The strict classifier alone would never clear debt on the 43/49 recorded runs that verify via ad-hoc scripts, and a debt-based nudge would misfire on tasks that pass today. Samples carry discriminating/debt_age into trajectories; e2ebench condenses them to TTFDC (run start to first discriminating observation) and debt-age max per run, aggregated on the Outcome-shadow report line. Telemetry only - the conservative two-blind-mutations nudge waits for this data.
The intervention phase's first runtime policy. Trigger: verification debt open AND blind_mutations >= 3, at most once per turn. Calibrated against two recorded baseline runs: healthy trajectories peak at two blind mutations (edit, edit, check), so the threshold spares 92% of currently passing runs, while the one recorded failure peaked at five and would have been nudged at round 10 of 15 with half its runtime left. The nudge asks for the cheapest discriminating evidence and explicitly permits finishing the minimum coherent multi-file change first - the goal is feedback, not test-after-every-edit. Enforcement is env-gated (REASONIX_EXPERIMENT_EBM=1, e2ebench -policy ebm); eligibility is stamped on every arm so baseline carries the same shadow, keeping the A-vs-B comparison inside one dataset. Runtime records only blind_mutations/ebm_eligible/ebm_fired per round; e2ebench derives the whole mechanism chain offline: fire round, blind and debt at fire, rounds/ms/reasoning-tokens to the first discriminating check, compliance within 1 and 2 rounds, plus debt area and blind peak per run - the readouts that separate "nudge ignored" from "early evidence useless".
Twenty tasks in three a-priori classes, so the EBM trigger is measured for sensitivity AND specificity instead of one blended eligibility rate: - ebm-positive (10): scattered two-spot bugs with a cheap high-information oracle shipped in the workdir - early discriminating evidence should win. - ebm-neutral (5): coherent multi-file migrations (schema/enum/rename/ units/config) where nothing runs until the whole patch lands - a premature nudge must not derail correct work. - ebm-negative (5): no cheap oracle exists (doc contracts, prod-only migration, a 45s soak as the only check, style renames) - the nudge must not shove the model into worthless full checks. Every grader was validated in both directions before landing: the seeded workdir FAILS verify.sh and a reference solution PASSES it, so experiment verdicts cannot be artifacts of a broken oracle. Class labels ride the existing task.toml class field; per-class exposure falls out of the standard report. Usage: e2ebench -suite benchmarks/verification-stress [-policy ebm]
esengine
force-pushed
the
feat/verification-stress-suite
branch
from
August 8, 2026 08:52
da222b3 to
3491a80
Compare
Owner
Author
|
Suite v2 after the first per-class exposure measurement (the instrument worked exactly as intended): v1 exposure came back inverted — 0/10 positive, 5/5 neutral, 5/5 negative. Root cause: positive tasks shipped their oracle in the workdir and advertised it in the prompt, so the agent verified immediately and blind never reached 3 — the pathology EBM targets (verifiable-but-nobody-hands-you-the-test) never arose. v2: oracle files removed from positive workdirs, assertions inlined into the hidden graders, prompts describe symptoms only. Cheap discriminating checks remain constructible (a 3-line repro settles debt via the broad classifier). All 20 graders re-validated in both directions. v2 measured exposure (arm A): positive 6/10 · neutral 5/5 · negative 5/5, pass 9/10 on positive, TTFDC p50 18s → 26s — the temptation to defer verification is now real. All three experiment populations are live. |
github-actions
Bot
added
the
v2
github-actions
Bot
added
the
agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7952. The mechanism experiment's instrument: a suite where the EBM trigger's precision is measurable per class, not as one blended eligibility number.
Design
Per the experiment plan, headline metrics are per-class exposure rates (positive/neutral/negative) and the mechanism chain (blind peak, debt area, extra blind mutations after EBM, TTFDC) on paired eligible trajectories - not overall wall time.
Grader trust
Every task was validated in both directions before landing: the seeded workdir FAILS
verify.sh, and a reference solution PASSES it. One grader case was rewritten during validation because it tested an unspecified float rounding boundary (9.995-to-cents) - graders must not punish unspecified behavior.Usage
Documentation-impact: none - benchmark fixtures for a developer experiment; not covered by docs/*.md
Cache-impact: none - benchmark fixtures only; commits inherited from already-merged stack parents are content-identical to main-v2, so no prompt or tool bytes change
Cache-guard: existing progress-guard and fork payload-invariant tests pin all prompt bytes; this PR adds no runtime code