fix: avoid XADD-with-fetch atomic that aborts on older LLVM BPF backends

block_rq_issue used __sync_fetch_and_add's return value against a shared
BPF_ARRAY counter to assign req_id, which lowers to a BPF XADD with the
BPF_FETCH flag. Older/distro clang BPF backends can't legally emit that
and hard-abort the whole process with "LLVM ERROR: Invalid usage of the
XADD return value" (uncatchable as a Python exception, since BCC embeds
clang/LLVM in-process). Replace the shared atomic counter with a
BPF_PERCPU_ARRAY bumped via a plain per-CPU increment, folding the CPU id
into the high bits for cross-CPU uniqueness, removing the need for any
atomic fetch-and-add entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: raflybro

docs/COMPATIBILITY_FIXES.md

@@ -81,3 +81,38 @@ dump. The captured data includes:
 * **System specs**: OS/distribution, CPU, and memory. The IP-geolocation
   country lookup is intentionally skipped on this path so a slow/unreachable
   network can't delay or block the dump.
+
+## 5. `LLVM ERROR: Invalid usage of the XADD return value` on Older LLVM/BCC Builds
+
+### The Problem
+On some machines, loading the BPF program aborted the whole process outright:
+```
+Loading BPF program... \ LLVM ERROR: line 3018: Invalid usage of the XADD return value
+Aborted
+```
+No Python exception was raised — the `try/except` around BPF initialization
+never fired, because this is a hard `SIGABRT` from inside LLVM itself (BCC
+embeds clang/LLVM in-process rather than shelling out, so a fatal LLVM error
+crashes the whole interpreter).
+
+The cause was `prober.c`'s block-tracing code assigning a per-request id from
+`__sync_fetch_and_add(gen, 1)` against a single, shared (non-per-CPU)
+`BPF_ARRAY` counter. Because the *return value* of the atomic add was used
+(not discarded), clang had to lower it to a BPF `XADD` with the `BPF_FETCH`
+flag — a fetch-and-add variant that only newer LLVM BPF backends can legally
+emit. This project pins no LLVM/clang floor anywhere (only "Kernel: Linux
+5.4+" / "BCC: 0.18.0+" are documented, in `docs/TRACE_FORMAT.md`), and
+distros package BCC against whatever LLVM happens to be available, so the
+generated bytecode reliably crashed on older/other-distro toolchains while
+working fine on the machine used for development.
+
+### The Solution
+The shared atomic counter was replaced with a `BPF_PERCPU_ARRAY` counter: each
+CPU bumps its own copy with a plain (non-atomic) read-increment, and
+`bpf_get_smp_processor_id()` is folded into the high bits of the id so ids
+stay unique across CPUs. This removes the fetch-returning atomic entirely, so
+the compiler no longer needs `BPF_FETCH`/XADD support at all — sidestepping
+the LLVM-version dependency rather than trying to detect and gate around it.
+Nothing downstream depended on strict global monotonic ordering of `req_id`,
+only on per-request uniqueness/correlation, so the change is behavior-neutral
+for consumers.

src/tracer/prober/prober.c

@@ -676,10 +676,17 @@ struct block_rq_key_t {
 BPF_TABLE("lru_hash", struct block_rq_key_t, struct block_issue_ctx, block_start_times, 65536); /**< Issue time + submitter, keyed by dev+sector */
 BPF_TABLE("lru_hash", struct block_rq_key_t, u64, block_insert_times, 65536);  /**< Tracks block request insert time (queue latency) */
 
-/* Monotonic generator for per-request IDs (see block_event.req_id). A single
- * u64 bumped atomically at issue time; lets consumers disambiguate repeated
- * I/O to the same (dev, sector) and correlate a request across its lifecycle. */
-BPF_ARRAY(block_req_id_gen, u64, 1);
+/* Per-CPU generator for per-request IDs (see block_event.req_id): each CPU
+ * bumps its own counter (no cross-CPU race, so no atomic op is needed) and
+ * the CPU id is folded into the high bits so ids stay unique across CPUs.
+ * A shared counter bumped via __sync_fetch_and_add's return value was tried
+ * first, but that lowers to a BPF XADD-with-fetch, which older/distro clang
+ * BPF backends can't legally emit and crash on (LLVM ERROR: Invalid usage of
+ * the XADD return value) -- see docs/COMPATIBILITY_FIXES.md #5. Consumers
+ * only need per-CPU-monotonic ids to disambiguate repeated I/O to the same
+ * (dev, sector) and correlate a request across its lifecycle; nothing relies
+ * on strict global ordering across CPUs. */
+BPF_PERCPU_ARRAY(block_req_id_gen, u64, 1);
 
 /* Block-tracing diagnostics counters (per-CPU, summed in userspace at exit):
  *   [0] requests issued, [1] requests completed (emitted),
@@ -2884,8 +2891,10 @@ TRACEPOINT_PROBE(block, block_rq_issue) {
   // Assign a monotonic per-request id, carried to the completion event.
   u32 gen_key = 0;
   u64 *gen = block_req_id_gen.lookup(&gen_key);
-  if (gen)
-    ictx.req_id = __sync_fetch_and_add(gen, 1);
+  if (gen) {
+    u64 local_id = (*gen)++;
+    ictx.req_id = ((u64)bpf_get_smp_processor_id() << 48) | (local_id & 0xFFFFFFFFFFFFULL);
+  }
 
   block_start_times.update(&key, &ictx);