feat(bench): per-round cognition telemetry in trajectory digests - #7943
Conversation
A local dashboard for watching a bench run while it writes trajectories: 'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory per poll behind /api/state and serves one self-contained dark instrument page. The run renders as a strip of task segments; each task as a recessed per-round trace whose win/break polarity is carried by geometry (up/down off the baseline) because the green/red pair alone fails CVD validation. The renderer is incremental - DOM nodes are keyed, new rounds pop in, numbers tween, hover survives polling. A replay engine scrubs the whole run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a finished run can be watched back round by round. Suite awareness lists queued tasks as ghosts and prices the remaining time. summarizeTrajectory splits into scanTrajectoryFile + finish so the dashboard can read the raw outcome series the summary discards; rounds gain their observation timestamp for the replay timeline.
ee68089 to
28a30a8
Compare
The p90 tail decomposed into pure model-generation time, but the digest could not say what those seconds bought. Executor reasoning/completion tokens now join each model round: every classified round carries a roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions), totals accumulate per run, and rounds whose gap crosses 8s enter a slow-round census. Delegation rounds get their own classification - previously they read as mutations, hiding the cost story of a subagent call inside a bugfix. The report gains one Cognition line: totals, reasoning/solved, the output rate p50/p90 (uniform rates indict token volume, not serving), the slow-round share of model time, and delegation cost. Backfills over already-recorded trajectories - usage events have carried reasoning and completion tokens all along. Interval math moves to intervals.go to keep trajectory.go under the file-size ceiling.
28a30a8 to
cba08c0
Compare
|
Self-review pass over the stack found one real defect here, now fixed in the amended commit: Slow-round census counted recovery/compaction-tainted gaps. A 30s provider-retry gap would have entered the census as a "slow thinking round" carrying near-zero reasoning tokens, misattributing adapter flakiness as cognition. The census now requires the gap to be untainted ( Also verified during review: executor-only token attribution (planner/subagent/sidecar usage excluded before accumulation), digest/actions ordering via the batch's dispatch-order names slice, and the delegation round class staying inside productiveOutcomes so existing waste metrics keep their meaning. |
Stacked on #7941. PR 1 of the cognition-efficiency series: pure measurement, no runtime change.
Why
The 49-task tail decomposed into pure model-generation time (0 retries, 0 compactions, ~99% cache), but the trajectory digest could not say what a 29s round bought. Usage events have carried
reasoningTokens/completionTokensall along - this joins them per round so slow rounds become attributable and the numbers backfill over every already-recorded trajectory.What
Each classified model round now emits a
roundDigest: gap ms, tool ms, executor reasoning/completion/prompt tokens, and the round's actions. Executor-only - planner and subagent usage stay in their own buckets.Slow-round census: rounds with a gap >= 8s accumulate count/gap/reasoning totals per run.
delegationbecomes its own round classification (task/parallel_tasks/fleet/research). Previously such rounds classified asmutation, hiding an 82s research call inside a bugfix's round ledger. Counted productive, so waste metrics keep their meaning.One new report line in suite reports and
-mode traj:**Cognition** (49 recorded runs): **reasoning** 38,720 tok · **completion** 108,773 tok (806 reasoning/solved) · **output rate** p50 138 · p90 393 tok/s · **slow rounds** (>=8s) 11 = 22% of model time, 3,852 reasoning tok · **delegation** 2 rounds (1m23s in subagents)(real backfilled output over the recorded 49-task run)
intervals.goextracted fromtrajectory.go(interval math, unchanged) to stay under the file-size ceiling.Verification
go test ./cmd/e2ebench/(new fixture test: per-round token join, subagent exclusion, delegation classification, slow census, render line) - all passgofmt,go vet,repolintcleanDocumentation-impact: none - e2ebench is a developer benchmarking tool; the new report line and JSON fields are self-describing and not covered by docs/*.md