feat(agent): shadow admission control for research delegation - #7946
Merged
Conversation
A local dashboard for watching a bench run while it writes trajectories: 'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory per poll behind /api/state and serves one self-contained dark instrument page. The run renders as a strip of task segments; each task as a recessed per-round trace whose win/break polarity is carried by geometry (up/down off the baseline) because the green/red pair alone fails CVD validation. The renderer is incremental - DOM nodes are keyed, new rounds pop in, numbers tween, hover survives polling. A replay engine scrubs the whole run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a finished run can be watched back round by round. Suite awareness lists queued tasks as ghosts and prices the remaining time. summarizeTrajectory splits into scanTrajectoryFile + finish so the dashboard can read the raw outcome series the summary discards; rounds gain their observation timestamp for the replay timeline.
The p90 tail decomposed into pure model-generation time, but the digest could not say what those seconds bought. Executor reasoning/completion tokens now join each model round: every classified round carries a roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions), totals accumulate per run, and rounds whose gap crosses 8s enter a slow-round census. Delegation rounds get their own classification - previously they read as mutations, hiding the cost story of a subagent call inside a bugfix. The report gains one Cognition line: totals, reasoning/solved, the output rate p50/p90 (uniform rates indict token volume, not serving), the slow-round share of model time, and delegation cost. Backfills over already-recorded trajectories - usage events have carried reasoning and completion tokens all along. Interval math moves to intervals.go to keep trajectory.go under the file-size ceiling.
esengine
force-pushed
the
feat/round-cognition-telemetry
branch
from
August 8, 2026 07:51
28a30a8 to
cba08c0
Compare
One research call cost 65% of the worst benchmark task's wall clock - a config-format bugfix delegating to an external-research subagent. The boundary: local mutation-shaped turns get no expensive research unless the user asked for it, the call cites an external source, or the turn is not a local fix at all. Shadow means observed, never enforced: each gated call records a content-free verdict (tool, verdict, reason, turn intent) through a new sink capability into the trajectory. The turn text judged is the bounded classifier source the delivery gates already store, so the shadow and the gates read the same words. e2ebench joins verdicts with delegation tool durations (order- independent - the verdict lands after the result) and reports one line: gated calls, would-deny rate, and the subagent time behind denied calls. A/B enforcement waits for this shadow data.
esengine
force-pushed
the
feat/delegation-admission-shadow
branch
from
August 8, 2026 07:53
ee68089 to
cb326e5
Compare
Owner
Author
|
Self-review notes (no code change needed): the verdict is judged against the bounded classifier text the delivery gates already store, so admission and gates read the same words; audits are content-free enums; blocked-by-permission calls still record a verdict, which is intentional (the shadow measures the model's intent to delegate, not execution). Live-chain validated in a real |
esengine
changed the base branch from
feat/round-cognition-telemetry
to
main-v2
August 8, 2026 14:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7943. PR 2 of the cognition-efficiency series: shadow only, nothing is blocked.
Why
In the recorded 49-task run, one
researchdelegation on a config-format bugfix cost 82.1s of subagent time plus ~35s of surrounding model gaps - about 65% of the slowest task's wall clock, on a turn that needed no external information. Before enforcing anything, this PR measures how often a conservative admission boundary would have refused such calls, and what it would have reclaimed.What
delegationAdmission(internal/agent/delegation_admission.go): a pure verdict for each gated delegation call. Local mutation-shaped turns (viataskintent.Classify, judging the same bounded classifier text the delivery gates already store) getdeny - local_fix_no_external_needunless the user explicitly asked for research, the call's arguments cite an http(s) source, or the turn is not a local fix. Gated set:researchonly, per the data.New content-free sink capability
RecordDelegationAdmission(tool/verdict/reason/intent enums, never query text), forwarded through every capability-forwarding decorator and appended to trajectories asdelegation_admissionrecords.Verdicts are recorded post-batch alongside the existing shadow observers (
applyBatchGuards); zero behavior change, and the run-loop stays exactly its former size (the turn text is reused, not re-stored).e2ebench joins verdicts with delegation tool durations (order-independent: the verdict record lands after the tool result) and reports:
**Delegation admission** (shadow): N gated calls · would deny M (P%) · subagent time behind denied calls XsVerification
go testacross e2ebench, agent, trajectory, event, notify, telemetry, stats, control, cli, boot, tool/builtin - all pass;gofmt/go vet/repolintclean.Cache-impact: none - the provider-visible prefix and all prompt bytes are unchanged; the shadow only observes tool calls and emits sink audits
Cache-guard: existing progress-guard and storm tests pin injected texts; TestObserveDelegationAdmissionRecordsOnlyGatedTools proves the shadow emits audits without touching results
Documentation-impact: none - shadow measurement is invisible in the product; the trajectory record and e2ebench report line are developer-facing and not covered by docs/*.md