Bound BufferingRepositorySubscriber's debounce buffer (APP-4829)

[warp-agent-staging]

Description

Fixes a memory-usage bug: BufferingRepositorySubscriber::on_files_updated (crates/repo_metadata/src/repository.rs) coalesces every incoming filesystem-watcher RepositoryUpdate into BufferState.pending via merge_repository_updates, but the debounce flush loop had no maximum buffer size or age. Under sustained filesystem churn (a large git checkout, npm install, or a very active watched tree), the flusher never observes a quiet period, so pending.added/modified/deleted (HashSet<TargetFile>) and pending.moved (HashMap<TargetFile, TargetFile>) grow unbounded before ever being flushed. This is the dominant contributor in a freshly symbolicated macOS heap profile (Sentry event 4836b25da6904993af8aec96dfa45b5d on issue 7259255054, 10.63 GB sampled, 95.2% attributed to this call path), and is fix #1 from APP-4829.

This PR only implements fix #1 (bounding the debounce buffer). The other two facets described in APP-4829 are out of scope here: the AI codebase-index path-set clone was already fixed by #13413, and the file-tree-store clone is tracked separately by APP-4822.

Fix

BufferingRepositorySubscriber now bounds the debounce buffer with these cooperating pieces:

• The merge itself is bounded, not just the flush. Each incoming RepositoryUpdate is merged into pending one entry at a time, in the same phase order merge_repository_updates already applies (moves, adds, modifies, deletes, then the boolean flags). The bound is checked after every single entry, so pending never grows past max_pending_entries (currently 10,000) even when one incoming update by itself is far larger than the bound (e.g. a single very large batch from a huge checkout).
• Delivery is serialized and never dropped. inner.on_files_updated is only ever called for one batch at a time; a completion callback drains whatever's next before starting the next delivery.
• Batches formed while a delivery is in flight are coalesced, not queued. Instead of storing each such batch separately (which would still be unbounded -- it would just move the growth from pending into a queue, and duplicate the same paths across multiple stored batches), they are merged (via RepositoryUpdate::merge, a new public wrapper around the same merge_repository_updates) into a single accumulator (BufferState::next_delivery). Delivered as one batch once the in-flight delivery finishes.
• The one production consumer no longer drops updates while busy. RepoOutlines (app/src/ai/outline/native.rs) used to silently discard a filesystem update that arrived while a previous outline recomputation was still Pending. It now merges such updates into a single accumulator (OutlineState::pending_update, via RepositoryUpdate::merge) and applies it as one follow-up recomputation once the in-flight one completes.
• Unsubscribing releases outstanding work instead of leaking or delivering it late. on_unsubscribe now marks the buffer inactive and drops both pending and any coalesced backlog; an already-in-flight delivery's completion becomes a no-op instead of continuing to call inner (and keeping the buffer's memory alive) after the subscription has ended.
• Fixed a pre-existing correctness bug in merge_repository_updates that coalescing made reachable. The moves loop used return instead of continue when a move's source collapsed into an already-recorded add or modify, which would abandon every remaining move, add, modify, delete, and flag in that same call. Coalescing now calls merge_repository_updates with real, multi-entry batches on the path sustained churn actually exercises (merging a newly-bounded batch into the backlog while a delivery is in flight), so this was no longer a theoretical, narrow-window pre-existing issue -- it was a live way for the losslessness this PR promises to be violated. Fixed by changing both returns to continues.

What this bound does and does not guarantee (please read before assuming more than the code delivers): the per-entry merge plus coalescing collapses the backlog from O(events × paths-per-event) down to O(distinct un-applied paths) -- the minimum that can be retained without dropping anything. This is not an absolute cap. If the consumer never catches up during one sustained burst (the pathological case the ticket describes), the coalesced accumulator can still grow to approach the number of distinct changed paths in that burst, same as pending briefly can before its first bounded batch is cut. Only the first batch formed while nothing is in flight is bounded to exactly max_pending_entries; everything that arrives while that batch is being delivered is coalesced into one (unbounded-in-size, bounded-in-content-by-distinct-paths) backlog. A hard cap on that backlog would require dropping per-path fidelity -- an explicit dirty-state/rescan protocol -- which changes observable outline behavior and is explicitly out of scope for this PR; it should be a separate, human-reviewed follow-up if the O(distinct paths) behavior still isn't good enough in practice.

Why 10,000 for the one guaranteed-bounded batch: it keeps that first batch's content to a few tens of MB (each buffered entry clones a PathBuf), while staying large enough that ordinary bursts (tens-to-thousands of files) still coalesce normally through the debounce.

Accepted behavior change, not a defect: splitting a debounce window into more, earlier flushes means an add-then-delete (or move-then-delete-of-target) of the same path that straddles a forced-flush boundary no longer cancels out within one batch the way it would have if the whole window had coalesced together. The end state converges to the same thing; a subscriber just sees slightly more intermediate churn. This is inherent to forcing early flushes and isn't fixed here.

Linked Issue

This is tracked in Linear, not a GitHub issue: APP-4829.

Testing

• forced_flush_drains_pending_before_debounce_timer_fires: with an hours-long debounce and a small max_pending_entries override, feeds exactly enough updates to cross the threshold and asserts the flush is observed -- proving it came from the forced path, not the debounce.

• single_incoming_update_exceeding_the_bound_is_delivered_without_loss: feeds a single RepositoryUpdate far larger than the configured bound in one notify_subscriber call; asserts the very first delivered batch is exactly bounded, more than one batch is produced, and the union of every batch covers every entry exactly once with no duplicates. Uses a deterministic receive timeout so a regression fails fast instead of hanging until nextest's slow-timeout.

• forced_and_debounced_flushes_apply_every_update_exactly_once_with_a_slow_consumer: a slow mock consumer (that itself asserts it's never invoked again before its previous call's future resolved) drains forced and debounce-triggered batches; asserts the union of everything delivered accounts for every fed update exactly once, with no duplicates, and that nothing else arrives afterward. Same deterministic timeout.

• unsubscribe_cancels_pending_delivery_and_releases_the_backlog: unsubscribes while one batch is in flight and a second is only backlogged; asserts the in-flight batch still completes normally but the backlogged one is never delivered late.

• merge_repository_updates_applies_every_entry_when_a_move_collapses_into_added / ..._into_modified: direct unit tests pinning the return-vs-continue fix for both collapse branches -- confirmed these fail without the fix and pass with it.

• coalescing_a_move_that_collapses_into_an_existing_add_still_applies_later_entries: an end-to-end regression through the actual reachable path -- accumulates a backlog while a delivery is in flight, then coalesces a second batch whose move collides with the backlog's content, and asserts everything after that move in the second batch is still applied. Confirmed this fails without the fix and passes with it.

• concurrent_update_while_pending_is_merged_not_dropped_or_double_recomputed (in app/src/ai/outline/native_tests.rs): drives a real RepoOutlines recomputation (via ai::index::build_outline against a temp directory) and submits a second update while the first is still Pending; asserts it's merged into the accumulator rather than dropped or starting a second overlapping recomputation, then drives to completion and asserts both changes are reflected in the final outline's file count.

• cargo nextest run -p repo_metadata (148 passed, 2 skipped), cargo nextest run -p warp -E 'test(outline)' (the new outline test passes, confirmed not OOM-killed), cargo clippy -p repo_metadata --all-targets --tests -- -D warnings, cargo clippy -p warp --all-targets --tests -- -D warnings, cargo test -p repo_metadata --doc, cargo check -p warp --tests, and ./script/format --check all pass, with no OOM observed running any of these in this environment.

• Not manually tested with ./script/run: this is a background buffering/consumption behavior with no UI surface, exercised entirely by the unit tests above.

• The linked issue is labeled ready-to-spec or ready-to-implement. (N/A -- tracked in Linear, not a GitHub issue)

• Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes). (N/A -- no UI surface)

• I have manually tested my changes locally with ./script/run (N/A -- see Testing above)

Agent Mode

• Warp Agent Mode - This PR was created via Warp's AI Agent Mode

CHANGELOG-BUG-FIX: Fixed unbounded memory growth in the repository file watcher's debounce buffer under sustained filesystem churn (e.g. a large git checkout or npm install).

[warp-agent-staging]

This PR was generated with Warp.

Comment @warp-factory on this PR to send it follow-up work.

 

[warp-agent-staging]

Overview

This bounds BufferState.pending under sustained churn by merging incoming updates entry-by-entry and dispatching batches through a serialized delivery queue, and stops RepoOutlines discarding updates that arrive mid-recomputation. Posting findings rather than a verdict: the change is blocked on a design decision that needs a human, because the memory bound it adds is undone by the two unbounded queues it introduces.

Concerns

• The unbounded growth relocates rather than disappears. BufferState::delivery_queue and OutlineState::queued_updates are unbounded VecDeque<RepositoryUpdate>s, and under the exact churn APP-4829 describes, batches are produced faster than Outline::update can parse them, so a backlog of full batches accumulates where pending used to. This is the decision point: the consumer is genuinely slower than the producer, so memory cannot be bounded while keeping per-path fidelity for an arbitrarily large change set — either the queued work is coalesced into a bounded representation, or the pipeline needs an explicit bounded dirty-state/rescan protocol, and that second option changes observable outline behavior enough that it should not be chosen without a human.
• Two mechanical findings are gated on that decision and will be fixed once it is made, not left behind. on_unsubscribe aborts only the debounce flusher, so a queued delivery keeps its memory alive and can still fire after unsubscribe, and a never-resolving future leaves delivering stuck true. Separately, the new RepoOutlines retain-and-drain path has no direct coverage — the added test's mock resolves on full processing, whereas the real OutlineRepositorySubscriber resolves on channel enqueue, so the two-stage path that actually prevents the loss is untested.
• Coalescing across a forced cut is lossless but not identical. An add and a later delete of the same path now land in separate batches instead of cancelling, and a move followed by deletion of its target emits move+delete rather than a delete of the source. The end state is the same and the current production consumer is filesystem-only outline watching, so this is noted rather than blocking — but it is inherent to early flushing and worth knowing before another subscriber is added.

Verdict

Checks: build pass, tests pass (repo_metadata 144 passed, 2 skipped), CI green, visual proof n/a. The warp app-crate check could not be independently confirmed — it was killed on memory in the review sandbox, which is an environment limit and not a source diagnostic.

Found: 1 critical, 2 important, 0 suggestions, 0 nits

Responding as wilson: Open session · View factory task

[warp-agent-staging]

Fresh recurrence while this sat unmerged: Sentry event 60991a92, 2026-08-22, 9.8 GB sampled — 60% at the merge_repository_updates moves loop and 21% at the HashSet<TargetFile> growth, i.e. exactly the two sites this PR bounds. It's on v0.2026.07.22.09.01.stable_01, which predates this branch, so it's a build without the fix rather than evidence against it. Flagging as a priority signal for review.

Responding as wilson: Open session · View factory task