Skip to content

feat(agent): shadow admission control for research delegation - #7946

Merged
esengine merged 6 commits into
main-v2from
feat/delegation-admission-shadow
Aug 8, 2026
Merged

feat(agent): shadow admission control for research delegation#7946
esengine merged 6 commits into
main-v2from
feat/delegation-admission-shadow

Conversation

@esengine

@esengine esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Stacked on #7943. PR 2 of the cognition-efficiency series: shadow only, nothing is blocked.

Why

In the recorded 49-task run, one research delegation on a config-format bugfix cost 82.1s of subagent time plus ~35s of surrounding model gaps - about 65% of the slowest task's wall clock, on a turn that needed no external information. Before enforcing anything, this PR measures how often a conservative admission boundary would have refused such calls, and what it would have reclaimed.

What

  • delegationAdmission (internal/agent/delegation_admission.go): a pure verdict for each gated delegation call. Local mutation-shaped turns (via taskintent.Classify, judging the same bounded classifier text the delivery gates already store) get deny - local_fix_no_external_need unless the user explicitly asked for research, the call's arguments cite an http(s) source, or the turn is not a local fix. Gated set: research only, per the data.

  • New content-free sink capability RecordDelegationAdmission (tool/verdict/reason/intent enums, never query text), forwarded through every capability-forwarding decorator and appended to trajectories as delegation_admission records.

  • Verdicts are recorded post-batch alongside the existing shadow observers (applyBatchGuards); zero behavior change, and the run-loop stays exactly its former size (the turn text is reused, not re-stored).

  • e2ebench joins verdicts with delegation tool durations (order-independent: the verdict record lands after the tool result) and reports:

    **Delegation admission** (shadow): N gated calls · would deny M (P%) · subagent time behind denied calls Xs

Verification

  • New tests: verdict table (deny / user-requested / external-source / non-local-intent), gated-tools-only emission, trajectory record round-trip, e2ebench join + render.
  • go test across e2ebench, agent, trajectory, event, notify, telemetry, stats, control, cli, boot, tool/builtin - all pass; gofmt/go vet/repolint clean.

Cache-impact: none - the provider-visible prefix and all prompt bytes are unchanged; the shadow only observes tool calls and emits sink audits
Cache-guard: existing progress-guard and storm tests pin injected texts; TestObserveDelegationAdmissionRecordsOnlyGatedTools proves the shadow emits audits without touching results
Documentation-impact: none - shadow measurement is invisible in the product; the trajectory record and e2ebench report line are developer-facing and not covered by docs/*.md

A local dashboard for watching a bench run while it writes trajectories:
'e2ebench -mode serve -trajectories <dir>' re-summarizes every trajectory
per poll behind /api/state and serves one self-contained dark instrument
page. The run renders as a strip of task segments; each task as a recessed
per-round trace whose win/break polarity is carried by geometry (up/down
off the baseline) because the green/red pair alone fails CVD validation.

The renderer is incremental - DOM nodes are keyed, new rounds pop in,
numbers tween, hover survives polling. A replay engine scrubs the whole
run by wall-clock timestamp (play/pause, 30-600x, jump-to-task), so a
finished run can be watched back round by round. Suite awareness lists
queued tasks as ghosts and prices the remaining time.

summarizeTrajectory splits into scanTrajectoryFile + finish so the
dashboard can read the raw outcome series the summary discards; rounds
gain their observation timestamp for the replay timeline.
@esengine
esengine requested a review from SivanCola as a code owner August 8, 2026 07:17
@github-actions github-actions Bot added the agent Core agent loop (internal/agent, internal/control) label Aug 8, 2026
The p90 tail decomposed into pure model-generation time, but the digest
could not say what those seconds bought. Executor reasoning/completion
tokens now join each model round: every classified round carries a
roundDigest (gap, tool time, reasoning/completion/prompt tokens, actions),
totals accumulate per run, and rounds whose gap crosses 8s enter a
slow-round census. Delegation rounds get their own classification -
previously they read as mutations, hiding the cost story of a subagent
call inside a bugfix.

The report gains one Cognition line: totals, reasoning/solved, the output
rate p50/p90 (uniform rates indict token volume, not serving), the
slow-round share of model time, and delegation cost. Backfills over
already-recorded trajectories - usage events have carried reasoning and
completion tokens all along.

Interval math moves to intervals.go to keep trajectory.go under the
file-size ceiling.
@esengine
esengine force-pushed the feat/round-cognition-telemetry branch from 28a30a8 to cba08c0 Compare August 8, 2026 07:51
One research call cost 65% of the worst benchmark task's wall clock - a
config-format bugfix delegating to an external-research subagent. The
boundary: local mutation-shaped turns get no expensive research unless
the user asked for it, the call cites an external source, or the turn
is not a local fix at all. Shadow means observed, never enforced: each
gated call records a content-free verdict (tool, verdict, reason, turn
intent) through a new sink capability into the trajectory.

The turn text judged is the bounded classifier source the delivery
gates already store, so the shadow and the gates read the same words.
e2ebench joins verdicts with delegation tool durations (order-
independent - the verdict lands after the result) and reports one line:
gated calls, would-deny rate, and the subagent time behind denied
calls. A/B enforcement waits for this shadow data.
@esengine
esengine force-pushed the feat/delegation-admission-shadow branch from ee68089 to cb326e5 Compare August 8, 2026 07:53
@esengine

esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Self-review notes (no code change needed): the verdict is judged against the bounded classifier text the delivery gates already store, so admission and gates read the same words; audits are content-free enums; blocked-by-permission calls still record a verdict, which is intentional (the shadow measures the model's intent to delegate, not execution). Live-chain validated in a real reasonix run: an explicit research request produced {allow, user_requested, mutation} through telemetry→notify→trajectory. Known limitation, by design: the marker list for user_requested is a narrow V1 heuristic and can only produce false allows (never false denies matter in shadow mode). Rebased onto the amended #7943.

@esengine
esengine changed the base branch from feat/round-cognition-telemetry to main-v2 August 8, 2026 14:29
@github-actions github-actions Bot added the v2 Go rewrite (1.x) — main-v2 branch, active development label Aug 8, 2026
@esengine
esengine merged commit 16d95f8 into main-v2 Aug 8, 2026
22 checks passed
@esengine
esengine deleted the feat/delegation-admission-shadow branch August 8, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant