Skip to content

feat(compaction): report user turns the retention budget could not hold - #8424

Merged
esengine merged 1 commit into
main-v2from
feat/compaction-retention-observability
Aug 11, 2026
Merged

feat(compaction): report user turns the retention budget could not hold#8424
esengine merged 1 commit into
main-v2from
feat/compaction-retention-observability

Conversation

@esengine

Copy link
Copy Markdown
Owner

Follow-up to #8419, which made compaction keep folded user turns verbatim within
a budget. This closes the two gaps that PR left open.

The blind spot

Retention shipped with no way to see it fail. A turn past the budget still folds
into the digest, and the projection reads exactly the same whether every
constraint survived or one was dropped. So the single case where compaction
still hands a user's own words to the summarizer was the single case nobody
could observe — the same shape of failure the original change set out to fix,
one level up.

It also left [[keep]] useless in practice. An escape hatch documented only in
SPEC.md does not help the person who needed it at the moment they needed it.

Change

keepUserTurns returns a userTurnRetention count that rides the existing
telemetry path into CompactionTelemetry as user_kept / user_dropped, and a
committed checkpoint that had to drop a turn emits a warning naming
[[keep]]. The notice fires after commit and never on a rejected candidate — a
checkpoint that was not installed folded nothing, so warning there would be a
false alarm.

Two structural cleanups ride along, both in service of the change rather than
beside it:

  • The retention code moves to compact_user_turns.go. compact.go was at 705
    lines against the 800-line ceiling, and this is one responsibility.
  • partitionFoldForProjection drops its early / carried return values,
    which have been nil since content-driven summary replaced the early-user-turn
    hoist. Threading retention through would otherwise have made it five results.

Coverage

Effect test at the final boundary (internal/boot). Drives the real Build
assembly through an actual fold and asserts the constraint reaches the provider
request — with a summarizer whose digest deliberately records nothing, so a
digest cannot be what carries it. The constraint is the second user turn on
purpose: the first is pinned into the stable prefix and would not exercise
fold-region retention at all.

This test was verified by mutation, not just by passing: stubbing
keepUserTurns to return early turns it red with constraint absent from the post-compaction request. Restored and re-verified green.

Sub-agent inheritance. Sub-agents build from subagentOptions, the single
construction point task / read_only_task / parallel_tasks children share,
which already forwards ContextWindow, KeepPolicy and CompactRatio — so
retention is inherited with no new code. A test pins that inheritance and the
child-scoped budget, because a parent's instructions are the child's
constraints and nothing else in the child transcript records them.

Conservation law. partitionCoversRegion now also asserts that every user
turn in a region is either kept or counted as dropped. A turn can no longer go
missing from the accounting the way it previously could from the projection, and
every test using that helper inherits the check.

Verification

gofmt -l .        clean
golangci-lint     0 issues
repolint          113571 — byte-identical to the main-v2 baseline
go test ./...     all green, zero failures

Cache-impact: low - the stable system-prompt prefix is untouched; this adds counters to existing compaction telemetry and one post-commit notice. No change to which messages compose the projection, so the provider-visible bytes for a given transcript are identical to #8419.
Cache-guard: internal/boot/compaction_retention_effect_test.go asserts the post-compaction provider request through the real Build assembly (mutation-verified), plus the unchanged internal/agent/cachehit_e2e_test.go.
System-prompt-review: esengine - the only internal/boot change is a new test file; no system-prompt text, tool schema, memory content, or boot wiring is modified.
Documentation-impact: updated - docs/SPEC.md "What survives a fold" now states that a drop is reported rather than silent, with the matching text in docs/SPEC.zh-CN.md.

Retention landed with no way to see it fail. A turn past the budget still folds
into the digest, and the projection reads exactly the same either way — so the
one case where compaction hands a user's own words to the summarizer was the
one case nobody could observe. That also left [[keep]] useless in practice:
an escape hatch only helps someone told it exists at the moment it is needed.

keepUserTurns now returns a userTurnRetention count that rides the existing
telemetry to CompactionTelemetry (user_kept / user_dropped), and a committed
checkpoint that dropped a turn emits a warning naming [[keep]]. The notice
fires after commit, never on a rejected candidate: a checkpoint that was not
installed folded nothing.

The retention code moves to compact_user_turns.go — compact.go was 705 lines
against the 800 ceiling, and this is one responsibility. partitionFoldForProjection
drops its early/carried return values, which have been nil since content-driven
summary replaced the hoist, rather than growing to five results.

Coverage follows the two gaps this series left open:

- A boot-level effect test drives the real Build assembly through a fold and
  asserts the constraint reaches the provider request, with a summarizer whose
  digest deliberately records nothing. Verified by mutation: stubbing
  keepUserTurns to a no-op turns it red.
- Sub-agents inherit retention through subagentOptions, the single construction
  point they share; a test pins that inheritance and the child-scoped budget,
  since a parent's instructions are the child's constraints and nothing else in
  the child transcript records them.

partitionCoversRegion additionally asserts the conservation law: every user turn
in a region is either kept or counted as dropped, so a turn cannot go missing
from the accounting the way it previously could from the projection.
@esengine
esengine requested a review from SivanCola as a code owner August 11, 2026 17:13
@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development agent Core agent loop (internal/agent, internal/control) config Configuration & setup (internal/config) labels Aug 11, 2026
@esengine
esengine merged commit 0d2494d into main-v2 Aug 11, 2026
24 checks passed
@esengine
esengine deleted the feat/compaction-retention-observability branch August 11, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) config Configuration & setup (internal/config) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant