Skip to content

feat(evidence): shadow outcome scorer with trajectory recording and bench metrics - #7937

Merged
esengine merged 1 commit into
main-v2from
feat/progress-shadow-scorer
Aug 8, 2026
Merged

feat(evidence): shadow outcome scorer with trajectory recording and bench metrics#7937
esengine merged 1 commit into
main-v2from
feat/progress-shadow-scorer

Conversation

@esengine

@esengine esengine commented Aug 8, 2026

Copy link
Copy Markdown
Owner

PR 1 of the outcome-progress series: pure measurement, no behavior change.

Why

The adaptive progress guard (#7915) scores novelty - new reads, new commands, mutations. That conflates motion with progress: an agent editing a different file each round earns +3 forever while getting no closer to a passing check, and a delegation return earns +2 whether or not it produced usable evidence. Before changing the policy, this PR makes the gap measurable.

What

  • evidence.OutcomeTracker: a shadow scorer observing the same per-round receipts as the live ProgressTracker, decomposed by outcome instead of novelty:
    • exploration - first reads, first commands/actions, first failures (localization), delegation returns
    • verification - attempts of verification-classified commands (IsDeliveryVerificationCommand, the same single classifier readiness uses)
    • objective - a verification command turning fail->pass
    • regression - pass->fail
    • churn - successful mutations (a state transition, not proof of progress)
    • legacy_gain - the live scorer's verdict on the same receipts, so offline analysis can price its claims without replaying
  • New optional sink capability RecordOutcomeProgress (mirrors the contract-shadow pattern from feat(taskcontract): unified task record assembled from existing signals #7927); the trajectory recorder appends each sample as an outcome_progress record. Pass-through added to every capability-forwarding sink decorator.
  • Fixed pre-existing pass-through gaps found while wiring: notify.Sink dropped readiness audits, turn completions and contract-shadow audits; the telemetry sink dropped contract-shadow audits. With notifications or telemetry enabled, those records never reached the trajectory recorder.
  • e2ebench condenses the series per run into an outcome digest: false_progress_rounds (legacy gain never redeemed by an objective transition within 3 rounds), solution_stall_max (armed only once the run enters its solution phase, so research turns are not punished), best-vs-final verified state, regressed_from_best, and search_regret_ms. Suite reports and -mode traj gain one Outcome shadow aggregate line.
  • Recordings that predate the shadow scorer backfill from shell verification receipts (execution.verification on tool results), so existing trajectory corpora can already answer how many runs peaked above their final verified state - the number that decides how much PR 2 (best-known-state) and PR 3 (conservative rollback) are worth.

Verification

  • go test ./internal/evidence/ ./internal/event/ ./internal/trajectory/ ./internal/agent/ ./internal/notify/ ./internal/telemetry/ ./internal/stats/ ./internal/control/ ./internal/cli/ ./cmd/e2ebench/ ./internal/tool/builtin/ ./internal/boot/ - all pass
  • TestOutcomeShadowRecordsEveryRoundWithoutTouchingGuards proves guard-visible round outputs stay byte-identical with the shadow active
  • gofmt, go vet, repolint clean

Cache-impact: none - the provider-visible prefix and all injected guard texts are byte-identical; the agent change is a private struct field plus a sink capability call after each batch
Cache-guard: TestOutcomeShadowRecordsEveryRoundWithoutTouchingGuards plus the existing progress-guard tests pin the guard's injected texts; no prompt bytes are produced by the shadow path
Documentation-impact: none - shadow measurement is invisible in the product; the trajectory file gains an optional record type and e2ebench (a developer tool) one report line, neither covered by docs/*.md

…ench metrics

The progress guard scores novelty: new reads, new commands, mutations. That
conflates motion with progress - an agent editing different files each round
scores +3 forever while getting no closer to a passing check. This adds the
measurement layer for an outcome-based successor without changing behavior.

evidence.OutcomeTracker shadows ProgressTracker over the same per-round
receipts and decomposes them: exploration (novel information), verification
(classified attempts), objective (a verification command turning fail->pass),
regression (pass->fail), churn (mutations - state transitions, not progress
proof), plus the legacy scorer's gain for offline comparison. Delegation
returns are exploration at best; bookkeeping moves nothing.

Samples ride a new optional sink capability (RecordOutcomeProgress) into the
trajectory recorder; pass-through added on every capability-forwarding
decorator. The notify and telemetry sinks were silently dropping existing
audit capabilities (readiness, turn completion, contract shadow) - fixed
alongside, since the shadow channel dies at the same gaps.

e2ebench condenses the series per run: false-progress rounds (legacy gain
never redeemed by an objective transition within 3 rounds), solution stall,
best-vs-final verified state, and search regret. Recordings that predate the
scorer backfill from shell verification receipts, so existing trajectories
already answer how many runs peaked above their final state. Suite reports
and traj mode gain one Outcome-shadow line.
@esengine
esengine requested a review from SivanCola as a code owner August 8, 2026 05:58
@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development agent Core agent loop (internal/agent, internal/control) labels Aug 8, 2026
@esengine
esengine merged commit 8240402 into main-v2 Aug 8, 2026
23 checks passed
@esengine
esengine deleted the feat/progress-shadow-scorer branch August 8, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant