feat(evidence): verification-debt telemetry in the outcome shadow - #7947
Merged
Conversation
A local dashboard for watching a bench run while it writes trajectories: 'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory per poll behind /api/state and serves one self-contained dark instrument page. The run renders as a strip of task segments; each task as a recessed per-round trace whose win/break polarity is carried by geometry (up/down off the baseline) because the green/red pair alone fails CVD validation. The renderer is incremental - DOM nodes are keyed, new rounds pop in, numbers tween, hover survives polling. A replay engine scrubs the whole run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a finished run can be watched back round by round. Suite awareness lists queued tasks as ghosts and prices the remaining time. summarizeTrajectory splits into scanTrajectoryFile + finish so the dashboard can read the raw outcome series the summary discards; rounds gain their observation timestamp for the replay timeline.
The p90 tail decomposed into pure model-generation time, but the digest could not say what those seconds bought. Executor reasoning/completion tokens now join each model round: every classified round carries a roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions), totals accumulate per run, and rounds whose gap crosses 8s enter a slow-round census. Delegation rounds get their own classification - previously they read as mutations, hiding the cost story of a subagent call inside a bugfix. The report gains one Cognition line: totals, reasoning/solved, the output rate p50/p90 (uniform rates indict token volume, not serving), the slow-round share of model time, and delegation cost. Backfills over already-recorded trajectories - usage events have carried reasoning and completion tokens all along. Interval math moves to intervals.go to keep trajectory.go under the file-size ceiling.
One research call cost 65% of the worst benchmark task's wall clock - a config-format bugfix delegating to an external-research subagent. The boundary: local mutation-shaped turns get no expensive research unless the user asked for it, the call cites an external source, or the turn is not a local fix at all. Shadow means observed, never enforced: each gated call records a content-free verdict (tool, verdict, reason, turn intent) through a new sink capability into the trajectory. The turn text judged is the bounded classifier source the delivery gates already store, so the shadow and the gates read the same words. e2ebench joins verdicts with delegation tool durations (order- independent - the verdict lands after the result) and reports one line: gated calls, would-deny rate, and the subagent time behind denied calls. A/B enforcement waits for this shadow data.
esengine
force-pushed
the
feat/delegation-admission-shadow
branch
from
August 8, 2026 07:53
ee68089 to
cb326e5
Compare
The failed benchmark task spent 75s thinking and 0.5s in tools - edits without ever asking the environment whether the hypothesis held. The shadow scorer now tracks that pattern: a mutation opens verification debt, every round without a discriminating observation ages it, and a discriminating observation settles it. Discriminating is deliberately broader than delivery verification - a verification-classified command, or any command exercising a file a mutation touched this turn (the repro-script case). The strict classifier alone would never clear debt on the 43/49 recorded runs that verify via ad-hoc scripts, and a debt-based nudge would misfire on tasks that pass today. Samples carry discriminating/debt_age into trajectories; e2ebench condenses them to TTFDC (run start to first discriminating observation) and debt-age max per run, aggregated on the Outcome-shadow report line. Telemetry only - the conservative two-blind-mutations nudge waits for this data.
esengine
force-pushed
the
feat/verification-debt-telemetry
branch
from
August 8, 2026 07:54
2e851ce to
a532623
Compare
Owner
Author
|
Self-review found one real defect here, now fixed in the amended commit: Read-only inspection of a mutated file settled verification debt. Also verified: debt state resets with the turn evidence (tracker is rebuilt in resetTurnEvidence), TTFDC guards against zero timestamps, and the run-2 bench exercised the field end-to-end (41/49 runs discriminating, debt max 6). |
esengine
changed the base branch from
feat/delegation-admission-shadow
to
main-v2
August 8, 2026 14:29
github-actions
Bot
added
the
v2
github-actions
Bot
added
the
agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7946. PR 3 of the cognition-efficiency series: telemetry only, no behavior change.
Why
The one failed task in the recorded 49-run suite spent 75s of model time and 0.5s in tools - mutations without ever asking the environment whether the hypothesis held. Solution stall cannot see this (it also fires on healthy runs); verification debt names it precisely: how long does a mutation sit unverified.
What
OutcomeTrackergains a debt state machine: a mutation opens debt, each round without a discriminating observation ages it (mutation round included), a discriminating observation settles it. Per-round samples now carrydiscriminatinganddebt_age.IsDeliveryVerificationCommand(unchanged, still owns readiness), OR any command mentioning the basename of a file mutated this turn - the deterministic hook for repro scripts and targeted runs. The recorded corpus proves the strict tier alone cannot clear debt on 43/49 runs that verify via ad-hoc scripts, so a debt-based policy built on it would misfire on tasks that pass today.outcome_progressrecords carry both fields (additive keys).ttfdc_ms(run start to first discriminating observation; zero = never) anddebt_age_max, and the Outcome-shadow report line gains:**discriminating checks** in K/N runs (TTFDC p50 Xs) · **verification debt max** D rounds.Verification
go testacross evidence, e2ebench, trajectory, agent, tool/builtin, boot, cli - all pass;gofmt/go vet/repolintclean.Cache-impact: none - no prompt bytes change; the shadow only observes receipts and records samples
Cache-guard: TestOutcomeShadowRecordsEveryRoundWithoutTouchingGuards plus progress-guard tests pin all injected texts; the debt fields ride the existing sample channel
Documentation-impact: none - shadow telemetry is invisible in the product; trajectory fields and the e2ebench report line are developer-facing and not covered by docs/*.md