feat(evidence): shadow outcome scorer with trajectory recording and bench metrics - #7937
Merged
Conversation
…ench metrics The progress guard scores novelty: new reads, new commands, mutations. That conflates motion with progress - an agent editing different files each round scores +3 forever while getting no closer to a passing check. This adds the measurement layer for an outcome-based successor without changing behavior. evidence.OutcomeTracker shadows ProgressTracker over the same per-round receipts and decomposes them: exploration (novel information), verification (classified attempts), objective (a verification command turning fail->pass), regression (pass->fail), churn (mutations - state transitions, not progress proof), plus the legacy scorer's gain for offline comparison. Delegation returns are exploration at best; bookkeeping moves nothing. Samples ride a new optional sink capability (RecordOutcomeProgress) into the trajectory recorder; pass-through added on every capability-forwarding decorator. The notify and telemetry sinks were silently dropping existing audit capabilities (readiness, turn completion, contract shadow) - fixed alongside, since the shadow channel dies at the same gaps. e2ebench condenses the series per run: false-progress rounds (legacy gain never redeemed by an objective transition within 3 rounds), solution stall, best-vs-final verified state, and search regret. Recordings that predate the scorer backfill from shell verification receipts, so existing trajectories already answer how many runs peaked above their final state. Suite reports and traj mode gain one Outcome-shadow line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of the outcome-progress series: pure measurement, no behavior change.
Why
The adaptive progress guard (#7915) scores novelty - new reads, new commands, mutations. That conflates motion with progress: an agent editing a different file each round earns +3 forever while getting no closer to a passing check, and a delegation return earns +2 whether or not it produced usable evidence. Before changing the policy, this PR makes the gap measurable.
What
evidence.OutcomeTracker: a shadow scorer observing the same per-round receipts as the liveProgressTracker, decomposed by outcome instead of novelty:IsDeliveryVerificationCommand, the same single classifier readiness uses)RecordOutcomeProgress(mirrors the contract-shadow pattern from feat(taskcontract): unified task record assembled from existing signals #7927); the trajectory recorder appends each sample as anoutcome_progressrecord. Pass-through added to every capability-forwarding sink decorator.notify.Sinkdropped readiness audits, turn completions and contract-shadow audits; the telemetry sink dropped contract-shadow audits. With notifications or telemetry enabled, those records never reached the trajectory recorder.outcomedigest:false_progress_rounds(legacy gain never redeemed by an objective transition within 3 rounds),solution_stall_max(armed only once the run enters its solution phase, so research turns are not punished), best-vs-final verified state,regressed_from_best, andsearch_regret_ms. Suite reports and-mode trajgain one Outcome shadow aggregate line.execution.verificationon tool results), so existing trajectory corpora can already answer how many runs peaked above their final verified state - the number that decides how much PR 2 (best-known-state) and PR 3 (conservative rollback) are worth.Verification
go test ./internal/evidence/ ./internal/event/ ./internal/trajectory/ ./internal/agent/ ./internal/notify/ ./internal/telemetry/ ./internal/stats/ ./internal/control/ ./internal/cli/ ./cmd/e2ebench/ ./internal/tool/builtin/ ./internal/boot/- all passTestOutcomeShadowRecordsEveryRoundWithoutTouchingGuardsproves guard-visible round outputs stay byte-identical with the shadow activegofmt,go vet,repolintcleanCache-impact: none - the provider-visible prefix and all injected guard texts are byte-identical; the agent change is a private struct field plus a sink capability call after each batch
Cache-guard: TestOutcomeShadowRecordsEveryRoundWithoutTouchingGuards plus the existing progress-guard tests pin the guard's injected texts; no prompt bytes are produced by the shadow path
Documentation-impact: none - shadow measurement is invisible in the product; the trajectory file gains an optional record type and e2ebench (a developer tool) one report line, neither covered by docs/*.md