Skip to content

Add foreground-task spawn-site census for memory attribution (APP-5393) - #15208

Open
warp-agent-staging[bot] wants to merge 2 commits into
masterfrom
factory/app-5393-foreground-task-census
Open

Add foreground-task spawn-site census for memory attribution (APP-5393)#15208
warp-agent-staging[bot] wants to merge 2 commits into
masterfrom
factory/app-5393-foreground-task-census

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Description

This PR does not fix a memory leak. Investigation into Sentry issue 7259255054 (macOS Stable, phys_footprint alert) found no attributable leak:

  • The Sentry issue is a catch-all: ~4,117 events / ~2,833 users, open since 2026-02-11, with several distinct Linear issues already filed against it.
  • The alert fires on macOS phys_footprint, which includes compressed/swapped pages. Across the latest 100 events, ~53 had compressed memory at ≥70% of the reported footprint, so the headline 8-20 GB figures aren't straightforwardly live leaked heap.
  • The dominant call chain (DispatchDelegate::run_on_main_threadasync_task::Runnable::run → boxed dyn Future::poll) is shared by every foreground task. The stack can't name which task is responsible, and the leaf frames are unsymbolized AppKit.
  • The one genuinely attributable sibling on this same Sentry issue was already found and fixed: APP-5401 / a9c0a1e (Coalesce superseded ProjectContextModel rule refresh tasks (APP-5401) #15147), which coalesced piled-up ProjectContextModel::refresh_project_rules_for_repo tasks. It merged 2026-08-15, after the affected release (v0.2026.07.29.09.05.stable_02).

What this PR does build is the instrumentation the investigation called for (angle 3): a live census of in-flight Foreground (main-thread) tasks, keyed by their spawn call site, attached to the existing excessive-memory Sentry report. The next time this alert fires, the event will name the call sites holding tasks open instead of dead-ending in Future::poll.

To be explicit about what this does and does not do: it will not reduce memory usage. It explains it.

How it works

  • Foreground::spawn / spawn_abortable / spawn_boxed are #[track_caller], and a ForegroundTaskCensus (a plain RefCell-backed map, since the executor is main-thread only — no lock contention) increments a per-site counter on spawn and decrements it when the task completes or is dropped (covering abort/cancellation, via a guard).
  • Foreground::task_census_snapshot(limit) returns the total live task count plus the top-N spawn sites by live count, sorted descending.
  • Wired into app/src/system/info.rsapp/src/profiling.rs::dump_jemalloc_heap_profile, which now takes the snapshot and attaches it as a second foreground_task_census Sentry context, alongside the existing memory_breakdown context.

The #[track_caller] chain is the whole ballgame

#[track_caller] on the three Foreground methods alone is not enough, and getting this wrong is the one way this change silently does nothing useful. Application code doesn't call the executor directly; it goes through context wrappers, and a wrapper that drops the attribute becomes the recorded location for every one of its callers. Two separate collapse points had to be fixed:

  1. ctx.spawn(...) routes through AppContext::spawn_local — a single fixed line in app.rs. Left unannotated, every ctx.spawn call in the app would collapse into that one line. The full chain (ctx.spawnModelContext::spawn_abortableModelContext::spawn_localAppContext::spawn_localForeground::spawn_boxed) is now annotated, plus the ViewContext equivalents and the spawn_with_retry_on_error(_when) helpers. This is the path the refresh_project_rules_for_repo sibling used.
  2. Stream tasks (ctx.spawn_stream_local(...), ctx.spawner()) route through AppContext::spawn_stream_local. The ModelContext/ViewContext wrappers above it were still missing the attribute, so a whole class of potentially long-lived stream tasks collapsed onto two fixed lines in the context files. Caught in review and fixed in c48aa9e.

Because this is silent when wrong, both collapse points now have regression tests that fail without the attributes.

Feature gating

The counting itself (in warpui_core) is always compiled in and always on — it's cheap (a hash-map entry bump plus an Rc clone per spawn, no locks, no extra heap allocation since it reuses the caller's existing LocalBoxFuture box) and useful independent of Sentry reporting. Only the Sentry-facing wiring in app/ is gated behind the existing heap_usage_tracking feature, matching how memory_breakdown is already gated. I didn't add a new feature flag since this isn't a product-facing toggle — it reuses the existing profiling-feature convention.

Scope

Foreground only, per the issue — Background::spawn_boxed has the same shape but runs across multiple real OS threads, so census tracking there would need a thread-safe structure (mutex/dashmap), which is a meaningfully different (and costlier) change. Not worth it for an executor that isn't implicated in this issue.

Linked Issue

Linear: APP-5393

Testing

  • ./script/format and the clippy invocations from ./script/presubmit pass, including cargo clippy -p warpui_core --all-targets --all-features --tests -- -D warnings.
  • cargo test -p warpui_core --lib: 321 passed, 0 failed, 7 ignored.
  • Executor-level unit tests in crates/warpui_core/src/async/native/executor_tests.rs cover:
    • a spawn is attributed to the actual calling file/line, not the executor's own;
    • distinct call sites are tracked and ranked separately (many tasks from one site + few from another shows up as a clear, separately-attributed outlier — this is what would have caught the refresh_project_rules_for_repo pile-up had it existed before APP-5401);
    • the requested top-N limit is respected;
    • completing, aborting, and dropping a task handle all correctly decrement the live count (no leaked counts on cancellation).
  • Context-level regression tests in crates/warpui_core/src/core/mod_tests.rs cover the second collapse point: distinct callers of ModelContext/ViewContext::spawn_stream_local and spawner() must resolve to distinct sites. Verified these actually regress: with the #[track_caller] attributes reverted, both tests fail with every site attributed to context.rs and none to the calling file; with them, both pass.
  • Verified cargo check -p warp_tui still builds (the TUI shares this executor).
  • One gap, stated plainly: cargo check -p warp --features heap_usage_tracking could not be completed in the authoring environment — rustc was SIGKILLed by a sandbox memory ceiling on the warp crate, including at CARGO_BUILD_JOBS=1. That is a resource limit, not an observed compile error, and the revision touches only warpui_core (adding attributes, which are not signature changes). Still, the feature-gated app/ wiring deserves a CI confirmation before merge.
  • No UI change, so no visual verification.

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Adds a live census of in-flight main-thread (Foreground) tasks, keyed
by the #[track_caller] call site that spawned them, so that a heap
profile whose stack dead-ends in the executor's boxed Future::poll can
still name the code responsible.

- Foreground::spawn/spawn_abortable/spawn_boxed are now #[track_caller],
  and the attribute is propagated up through AppContext::spawn_local,
  ModelContext::{spawn,spawn_abortable,spawn_with_retry_on_error(_when)},
  and the equivalent ViewContext methods, so that application-level
  ctx.spawn(...) call sites are recorded instead of collapsing into the
  shared plumbing that forwards to the executor.
- ForegroundTaskCensus tracks live/total counts per spawn site in a
  RefCell (the executor is main-thread only), incrementing on spawn and
  decrementing on completion or drop (covering abort/cancellation).
- Foreground::task_census_snapshot() exposes the top-N spawn sites by
  live count plus the total live task count.
- Wires the snapshot into the existing excessive-memory Sentry report
  (app/src/system/info.rs, app/src/profiling.rs) as a second
  foreground_task_census context, alongside the existing
  memory_breakdown context, gated the same way (heap_usage_tracking).

This is attribution instrumentation only; it does not fix a leak. See
the PR description for the investigation that concluded no
attributable leak exists in Sentry issue 7259255054.

Co-Authored-By: Warp <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 16, 2026
@warp-agent-staging warp-agent-staging Bot added factory:wilson area:performance:memory Memory usage, allocation, leaks, and memory-bound performance. labels Aug 16, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

View run View conversation

The census added in the previous commit relies on an unbroken chain of
`#[track_caller]` from the application's call site down to
`Foreground::spawn_boxed`. `ModelContext`/`ViewContext::spawn_stream_local`
and `spawner` were missing the attribute, so every stream task in the app
collapsed onto a single line in the context wrapper -- the same bucketing
failure the census exists to prevent, which would have hidden a real
pile-up behind one entry.

The added tests fail without the attributes (every site resolves to
`context.rs`, so no site in the test file is recorded) and pass with them.

Co-Authored-By: Warp <agent@warp.dev>
@warp-agent-staging
warp-agent-staging Bot marked this pull request as ready for review August 19, 2026 12:15
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

Four more occurrences of this signature since this PR was opened (Sentry 7259255054, now 4,197 events): ~11.19 GB, ~12.02 GB, ~15.73 GB, ~7.13 GiB. Two were captured with warp.application_stage: Active, so this is not backgrounded-only as the ticket originally framed it.

All five affected releases (07.15, 07.22, 07.29, 08.12 stables) predate a9c0a1e, the related APP-5401 fix — so none of these events tests it, and this cluster is not evidence that it failed.

Marked ready for review to run the full CI suite, which was skipped while this sat in draft.

Responding as wilson: Open session · View factory task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:performance:memory Memory usage, allocation, leaks, and memory-bound performance. cla-signed factory:wilson

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants