Skip to content

APP-5495: Force GNU build-ids on Linux binaries so daemon pprof dumps can be symbolicated - #15265

Draft
warp-agent-staging[bot] wants to merge 2 commits into
masterfrom
factory/app-5495-daemon-pprof-build-id
Draft

APP-5495: Force GNU build-ids on Linux binaries so daemon pprof dumps can be symbolicated#15265
warp-agent-staging[bot] wants to merge 2 commits into
masterfrom
factory/app-5495-daemon-pprof-build-id

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes the memory-triage blocker for the Linux remote-server-daemon: its jemalloc heap-profile dumps had a degenerate mapping table (no address range, no GNU build-id), making them impossible to symbolicate offline. See Sentry issue 7259255054, event 2623254a94e8432c860a6e8a0a3000db, and Linear APP-5495.

Mechanism (established empirically, not by inference)

Two independent things are going on, and only one of them is actually broken:

  1. memory_start=0, memory_limit=u64::MAX, file_offset=0 on every Mapping record is intentional upstream behavior, not a bug, and not daemon-specific. jemalloc_pprof/pprof_util 0.8.x deliberately zero these fields for every mapping on every platform (see polarsignals/rust-jemalloc-pprof#26, "machine-independent-addrs") — sample addresses are pre-resolved to file-relative offsets before the proto is written, so the address range is redundant by design. I confirmed this by writing a standalone repro that links jemalloc_pprof 0.8.2 (the version this repo actually resolves), dumps a real pprof, and decodes the Mapping table directly: every mapping — main binary and shared libs alike, on a normal native build — has memory_start=0/memory_limit=u64::MAX/file_offset=0. This is consistent with APP-5350: the main client's mappings look "fine" only because their build_id is non-empty, not because their address range is real.
  2. The daemon's build_id is empty because it's missing a GNU build-id note, which is a real, daemon-specific bug. The daemon (built as the oz/CLI artifact, via script/linux/bundle --artifact cli) is cross-compiled for x86_64-unknown-linux-musl using the vendored musl-cross-make toolchain from script/linux/configure_musl_toolchain (cross-tools/musl-cross, tag 20250929, GNU ld 2.45). That linker's default is --build-id=none, unlike the Debian/Ubuntu-patched ld used for the main client's native Linux builds (which defaults build-id on). I confirmed this three ways:
    • Compiled a trivial C program with that exact toolchain: no default build-id note (readelf -n shows only .note.gnu.property); adding -Wl,--build-id=sha1 produces one.
    • Cross-compiled a standalone repro (same jemalloc_pprof 0.8.2, same musl-cross toolchain, no extra flags) that links tikv-jemallocator and dumps a real pprof. The resulting Mapping table has 4 entries, all with the same filename (the daemon binary itself — as a static musl binary, dl_iterate_phdr sees only itself, and each of its 4 PT_LOAD segments becomes one Mapping), all with empty build_id — this exactly reproduces the Sentry event's reported pattern. Two of these captured dumps (with and without a build-id) are committed as this PR's test fixtures.
    • Rebuilding the same repro with -Wl,--build-id=sha1 added produces a real, non-empty build-id in every Mapping entry, with the rest of the shape (still memory_start=0/memory_limit=u64::MAX) unchanged, confirming the build-id is the only thing that was actually broken.
    • Confirmed the native Linux build (x86_64-unknown-linux-gnu, this repo's actual .cargo/config.toml) already emits a build-id by default, both via a raw cc invocation and via a real cargo test build of the warp crate.

Fix

Added -Wl,--build-id=sha1 for every Linux link target in .cargo/config.toml ([target.'cfg(target_os = "linux")']), matching the existing macOS-specific rustflags entry there. This is the cheapest option from the ticket's suggested list and directly addresses the actual mechanism: it guarantees a build-id note on every Linux binary we ship, including the daemon/CLI, regardless of which linker built it. I did not attempt to "fix" the zeroed address ranges — that's deliberate upstream behavior, and reintroducing real (ASLR-dependent, non-reproducible) addresses into a format that no longer needs them would be a regression, not a fix.

Also added a warning (app/src/profiling.rs::warn_if_pprof_missing_build_ids) so a future degenerate dump — from this path or any other cause — shows up in logs (log::warn!) instead of only being discoverable by decoding an individual Sentry attachment by hand. It walks just enough of the pprof Profile protobuf to check each Mapping's build_id field, without a full protobuf/pprof decoder dependency.

Changes

  • .cargo/config.toml: force -Wl,--build-id=sha1 on all Linux link targets.
  • app/src/profiling.rs: dump_jemalloc_pprof_bytes() now inspects the dump it produces and logs a warning if any Mapping entries have no build-id; adds the minimal protobuf-walking helpers backing that check.
  • app/src/profiling_tests.rs + app/src/profiling_fixture_*: unit tests for the protobuf walker, backed by real jemalloc_pprof 0.8.2 pprof dumps captured from the standalone repro (one with a populated build-id, one without, plus a truncated fixture for the error path) rather than a hand-rolled protobuf encoder, so the tests exercise the actual wire format instead of only the test's own assumptions about it. Both the low-level parser and the gzip-decoding entry point are covered.

Verification

Verified:

  • A standalone reproduction (outside this repo, using the exact jemalloc_pprof 0.8.2 version this workspace resolves and the exact vendored musl-cross-make toolchain) reproduces the reported degenerate profile shape byte-for-byte (4 identical-filename mappings, memory_start=0/memory_limit=u64::MAX/file_offset=0, empty build_id), and confirms -Wl,--build-id=sha1 fixes the build-id. Two of its captured dumps are committed as this PR's test fixtures.
  • cargo test -p warp profiling:: (default features) and cargo test -p warp --features jemalloc_pprof,heap_usage_tracking profiling:: — all 6 unit tests pass in both cases.
  • cargo clippy -p warp --all-targets --tests -- -D warnings (the exact invocation script/presubmit uses for this crate) passes cleanly.
  • ./script/format run; no unrelated files touched.
  • Confirmed via readelf -n that a real cargo test build of the warp crate for the native x86_64-unknown-linux-gnu target still carries a build-id note after the .cargo/config.toml change (no regression there — it already had one).

Not verified — one check a human should run before merging: I did not produce the release-equivalent end-to-end proof — building the actual shipped daemon/CLI musl binary and inspecting it directly. A from-scratch build of the full warp crate for x86_64-unknown-linux-musl was not feasible in this environment (no cached build artifacts for that target, a very large dependency graph, and a real risk of the build being killed under memory pressure). Before merging, please run:

./script/linux/bundle --artifact cli -c dev
readelf -n target/x86_64-unknown-linux-musl/release-cli-debug_assertions/dev

(adjust the binary path/channel flag for whichever channel you build; script/linux/bundle --help lists the -c/--channel and profile options.)

A passing result shows a Displaying notes found in: .note.gnu.build-id section with a non-empty Build ID: ... line in the readelf -n output — mirroring what this PR's standalone repro already demonstrated with the identical toolchain and linker flag.

Linked Issue

  • Linear: APP-5495 (no linked GitHub issue).

Testing

  • I have manually tested my changes locally with ./script/run
    (Not applicable — this is a headless profiling/build-config fix with no UI surface. See Verification above for what was actually run, and for the one remaining check a human should run before merging.)

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

CHANGELOG-NONE

…gs (APP-5495)

Root cause: the daemon/CLI binary is cross-compiled for
x86_64-unknown-linux-musl with the vendored musl-cross-make toolchain
(script/linux/configure_musl_toolchain), whose linker does not enable a
GNU build-id note by default (unlike the distro-patched linkers used
for the main client's native builds). Without a build-id, jemalloc_pprof
heap profiles captured from the daemon have no way to join their
mapping table to a debug-info file offline.

- Force -Wl,--build-id=sha1 for every Linux link target in
  .cargo/config.toml, so all Linux binaries we ship carry a build-id.
- Log a warning from dump_jemalloc_pprof_bytes() when a captured pprof's
  mapping table has entries with no build-id, so a future degenerate
  dump is visible in logs instead of only discoverable by decoding an
  attachment.
- Add unit tests for the minimal protobuf walker backing that check.

Co-Authored-By: Warp <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 18, 2026
@warp-agent-staging warp-agent-staging Bot added factory:wilson area:performance:memory Memory usage, allocation, leaks, and memory-bound performance. labels Aug 18, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-factory on this PR to send it follow-up work.

View run View conversation

Review finding: the previous unit tests validated the pprof mapping
walker only against wire data produced by a hand-rolled encoder
defined in the same test file, so a shared wrong assumption between
encoder and parser could pass all tests while the production
diagnostic was still wrong.

Replace those fixtures with real jemalloc_pprof 0.8.2 dumps captured
from the standalone repro used to establish this PR's root cause: one
built natively (has a build-id) and one built with the repo's
vendored musl-cross-make toolchain without --build-id (reproducing the
daemon's pre-fix degenerate mapping table), plus a truncated fixture
for the error path. Also exercise the gzip-decoding entry point
(ungzip_and_count_pprof_mappings_missing_build_id) end to end, not
just the inner parser.

Co-Authored-By: Warp <agent@warp.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:performance:memory Memory usage, allocation, leaks, and memory-bound performance. cla-signed factory:wilson

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants