Skip to content

feat(bench): per-round cognition telemetry in trajectory digests - #7943

Merged
esengine merged 4 commits into
main-v2from
feat/round-cognition-telemetry
Aug 8, 2026
Merged

feat(bench): per-round cognition telemetry in trajectory digests#7943
esengine merged 4 commits into
main-v2from
feat/round-cognition-telemetry

Conversation

@esengine

@esengine esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Stacked on #7941. PR 1 of the cognition-efficiency series: pure measurement, no runtime change.

Why

The 49-task tail decomposed into pure model-generation time (0 retries, 0 compactions, ~99% cache), but the trajectory digest could not say what a 29s round bought. Usage events have carried reasoningTokens/completionTokens all along - this joins them per round so slow rounds become attributable and the numbers backfill over every already-recorded trajectory.

What

  • Each classified model round now emits a roundDigest: gap ms, tool ms, executor reasoning/completion/prompt tokens, and the round's actions. Executor-only - planner and subagent usage stay in their own buckets.

  • Slow-round census: rounds with a gap >= 8s accumulate count/gap/reasoning totals per run.

  • delegation becomes its own round classification (task/parallel_tasks/fleet/research). Previously such rounds classified as mutation, hiding an 82s research call inside a bugfix's round ledger. Counted productive, so waste metrics keep their meaning.

  • One new report line in suite reports and -mode traj:

    **Cognition** (49 recorded runs): **reasoning** 38,720 tok · **completion** 108,773 tok (806 reasoning/solved) · **output rate** p50 138 · p90 393 tok/s · **slow rounds** (>=8s) 11 = 22% of model time, 3,852 reasoning tok · **delegation** 2 rounds (1m23s in subagents)

    (real backfilled output over the recorded 49-task run)

  • intervals.go extracted from trajectory.go (interval math, unchanged) to stay under the file-size ceiling.

Verification

  • go test ./cmd/e2ebench/ (new fixture test: per-round token join, subagent exclusion, delegation classification, slow census, render line) - all pass
  • gofmt, go vet, repolint clean
  • Backfill exercised against the full recorded 49-task trajectory directory

Documentation-impact: none - e2ebench is a developer benchmarking tool; the new report line and JSON fields are self-describing and not covered by docs/*.md

A local dashboard for watching a bench run while it writes trajectories:
'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory
per poll behind /api/state and serves one self-contained dark instrument
page. The run renders as a strip of task segments; each task as a recessed
per-round trace whose win/break polarity is carried by geometry (up/down
off the baseline) because the green/red pair alone fails CVD validation.

The renderer is incremental - DOM nodes are keyed, new rounds pop in,
numbers tween, hover survives polling. A replay engine scrubs the whole
run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a
finished run can be watched back round by round. Suite awareness lists
queued tasks as ghosts and prices the remaining time.

summarizeTrajectory splits into scanTrajectoryFile + finish so the
dashboard can read the raw outcome series the summary discards; rounds
gain their observation timestamp for the replay timeline.
@esengine
esengine requested a review from SivanCola as a code owner August 8, 2026 07:06
@github-actions github-actions Bot added the agent Core agent loop (internal/agent, internal/control) label Aug 8, 2026
@esengine
esengine force-pushed the feat/round-cognition-telemetry branch from ee68089 to 28a30a8 Compare August 8, 2026 07:17
The p90 tail decomposed into pure model-generation time, but the digest
could not say what those seconds bought. Executor reasoning/completion
tokens now join each model round: every classified round carries a
roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions),
totals accumulate per run, and rounds whose gap crosses 8s enter a
slow-round census. Delegation rounds get their own classification -
previously they read as mutations, hiding the cost story of a subagent
call inside a bugfix.

The report gains one Cognition line: totals, reasoning/solved, the output
rate p50/p90 (uniform rates indict token volume, not serving), the
slow-round share of model time, and delegation cost. Backfills over
already-recorded trajectories - usage events have carried reasoning and
completion tokens all along.

Interval math moves to intervals.go to keep trajectory.go under the
file-size ceiling.
@esengine
esengine force-pushed the feat/round-cognition-telemetry branch from 28a30a8 to cba08c0 Compare August 8, 2026 07:51
@esengine

esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Self-review pass over the stack found one real defect here, now fixed in the amended commit:

Slow-round census counted recovery/compaction-tainted gaps. A 30s provider-retry gap would have entered the census as a "slow thinking round" carrying near-zero reasoning tokens, misattributing adapter flakiness as cognition. The census now requires the gap to be untainted (!gap.tainted && !gap.compaction); the round's gap itself stays booked in the round digest and outcome tallies, so nothing is hidden — it just no longer counts as a cognition purchase. Covered by TestSlowRoundCensusExcludesRecoveryGaps.

Also verified during review: executor-only token attribution (planner/subagent/sidecar usage excluded before accumulation), digest/actions ordering via the batch's dispatch-order names slice, and the delegation round class staying inside productiveOutcomes so existing waste metrics keep their meaning.

@esengine
esengine changed the base branch from feat/e2ebench-dashboard to main-v2 August 8, 2026 14:29
@github-actions github-actions Bot added the v2 Go rewrite (1.x) — main-v2 branch, active development label Aug 8, 2026
@esengine
esengine merged commit 86e796b into main-v2 Aug 8, 2026
22 checks passed
@esengine
esengine deleted the feat/round-cognition-telemetry branch August 8, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant