block: tag swap-origin (REQ_SWAP) requests in the flags column#95
Conversation
Swap I/O goes through the block layer with no filesystem counterpart, so on memory-constrained hosts it masquerades as filesystem block I/O, inflating apparent write amplification and polluting cacheability analysis. Tag it so analysis can subtract it cleanly (issue #94). The rwbs string never encodes REQ_SWAP and the block_rq_* tracepoint args carry no cmd_flags on mainline kernels, so the bit is read from struct request by a new best-effort kprobe on blk_mq_start_request() (the function that fires trace_block_rq_issue). Swap requests store a timestamped marker in a (dev, sector)-keyed LRU map; every other request scrubs the key. The completion handler honors a marker only when it is consistent with the matched issue context (no newer than the issue timestamp, no older than 1s, and a write op — the kernel sets REQ_SWAP only on swap writes), then emits is_swap, which userspace appends to the block flags column as a lowercase "swap" sub-flag alongside sync/meta/ahead. The BPF handler is compiled out on kernels without REQ_SWAP (< 4.19); KernelProbeTracker probes for it with load_func before attaching so such kernels do not pay a per-request kprobe trap for a no-op. If the symbol or handler is unavailable, block events simply carry no swap tag and nothing else is affected. Known limitations (documented in BLOCK_IO_EVENTS.md): swap-in reads are never tagged (the kernel only flags writes), and bio-based swap devices (e.g. zram) never appear in the block stream at all. Also aligns stale wording in BLOCK_IO_EVENTS.md with the code: rwbs sub-flags live in the flags column (not operation), the correlation key is a struct with separate dev/sector fields, and mainline tracepoints never exposed cmd_flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181MzEpBhHkug2Ftfg53G4L
Validates on a real kernel that trace_blk_mq_start_request compiles in (REQ_SWAP defined), passes the verifier, and attaches — a broken #ifdef guard would otherwise pass BPF() load silently and only surface at tracer startup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181MzEpBhHkug2Ftfg53G4L
There was a problem hiding this comment.
Code Review
This pull request introduces swap-origin tagging (the swap flag) for block I/O events to distinguish swap-out traffic from filesystem-origin block I/O. This is implemented via a best-effort BPF kprobe on blk_mq_start_request that captures the REQ_SWAP flag from struct request at issue time and propagates it to the completion event. Feedback recommends adding a defensive NULL check for the rq pointer in the new kprobe handler to prevent potential dereference issues.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| * guaranteed no-op. | ||
| */ | ||
| #ifdef REQ_SWAP | ||
| int trace_blk_mq_start_request(struct pt_regs *ctx, struct request *rq) { |
There was a problem hiding this comment.
For defensive programming and consistency with other probe entry points in prober.c, it is highly recommended to explicitly check if the rq pointer is NULL before performing any member accesses or dereferences.
int trace_blk_mq_start_request(struct pt_regs *ctx, struct request *rq) {
if (!rq)
return 0;There was a problem hiding this comment.
Declining this one: the kernel never calls blk_mq_start_request() with a NULL rq (the function unconditionally dereferences it), and under BCC every member access in a kprobe handler is rewritten into bpf_probe_read_kernel(), which fails safely on a bad pointer — disk would then read as NULL and the existing if (!disk) guard returns. An explicit if (!rq) would be unreachable code.
Generated by Claude Code
There was a problem hiding this comment.
Pull request overview
This PR adds swap-origin tagging to the Linux block I/O stream by appending a lowercase swap sub-flag into the existing free-form flags column (no schema/column additions), enabling downstream analysis to separate swap-out traffic from filesystem-origin block writes.
Changes:
- Add a best-effort kprobe on
blk_mq_start_request()to captureREQ_SWAPat request start and carry it toblock_rq_completevia an(dev, sector)-keyed marker map. - Merge the
is_swapmarker into the emitted blockflagsstring in userspace (FlagMapper.append_swap_flag) and add unit tests. - Update schema and block-trace documentation to describe the
swapflag and its caveats (swap-out only, best-effort availability, request-based devices only).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_flag_mapper.py | Adds unit tests covering swap flag merging behavior in the block flags pipeline. |
| src/tracer/FlagMapper.py | Adds append_swap_flag() helper to append swap into the block flags column. |
| src/tracer/IOTracer.py | Appends swap into the block flags output based on the emitted is_swap marker. |
| src/tracer/prober/prober.c | Emits an is_swap marker on block completions by correlating a REQ_SWAP marker captured at request start. |
| src/tracer/KernelProbeTracker.py | Attaches the new blk_mq_start_request kprobe best-effort (and skips it when unavailable). |
| src/tracer/schema.py | Updates the block stream flags column description to include swap. |
| docs/traces/BLOCK_IO_EVENTS.md | Documents swap-origin tagging mechanism and caveats; aligns operation/flags column semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| #if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 17, 0) | ||
| struct gendisk *disk = rq->q->disk; | ||
| #else | ||
| struct gendisk *disk = rq->rq_disk; | ||
| #endif | ||
| if (!disk) | ||
| return 0; | ||
| u32 dev = ((u32)disk->major << 20) | (u32)disk->first_minor; | ||
| struct block_rq_key_t key = block_rq_key(dev, rq->__sector); | ||
|
|
||
| if (rq->cmd_flags & REQ_SWAP) { | ||
| u64 ts = bpf_ktime_get_ns(); | ||
| block_swap_flags.update(&key, &ts); |
There was a problem hiding this comment.
This is already what happens under the hood: the tracer is BCC-based, and BCC's Clang rewriter converts member accesses on kprobe arguments into bpf_probe_read_kernel() calls automatically, one per hop of a multi-level chain (rq->q->disk included), compiled against the running kernel's headers. Direct-deref style is the file's existing idiom for kprobe handlers (e.g. file && file->f_inode in the VFS handlers); explicit bpf_probe_read_kernel() appears where the rewriter can't transform the access (array copies out of tracepoint args). Hand-written probe reads here would produce the same bytecode with more code. The BPF Compile CI job builds this handler as-is.
Generated by Claude Code
Closes #94.
Swap I/O goes through the block layer with no filesystem counterpart, so on memory-constrained hosts it masquerades as filesystem block I/O — inflating apparent write amplification and polluting block cacheability curves. This tags swap-origin requests with a lowercase
swapsub-flag in the block stream's existing free-formflagscolumn (alongsidesync/meta/ahead), so analysis can subtract it cleanly. No schema/column change.How it works
The issue's sketch needed two corrections found while implementing:
The bit cannot come from the
block_rq_issuehandler. A BCCTRACEPOINT_PROBEonly receives the tracepoint's declared fields (dev, sector, nr_sector, rwbs, comm) — there is nostruct request *inargson any kernel, and mainlineblock_rq_*tracepoints never carriedcmd_flagsat all (only patched vendor kernels do, per the repo's existingHAS_CMD_FLAGSsniffing). The CO-RE/BPF_CORE_READsuggestion also doesn't apply to this BCC-based tracer. Instead, a new best-effort kprobe onblk_mq_start_request()— the exported function that firestrace_block_rq_issue, 1:1 with the tracepoint on every kernel ≥ 5.0 — readsrq->cmd_flags & REQ_SWAPwith the running kernel's own headers supplying the correct bit position (it moves across versions). The issue'ssubmit_bioalternative was rejected: atsubmit_bioentry the sector is still partition-relative andbi_bdevis the partition, so(dev, sector)keys would never match the disk-absolute request-level events for swap partitions.REQ_SWAPtags swap-out only. Verified in mainline v5.15/v6.6/v6.12mm/page_io.c: the kernel setsREQ_SWAPexclusively on swap writes; swap-in reads are plainREQ_OP_READ. That is the half the motivating write-amplification analysis needs (pswpouttraffic), butpswpincorrelation is not possible with this mechanism on any kernel. Documented as a caveat.Data flow
(dev, sector)-keyed LRU map (block_swap_flags, sized like its siblings); every non-swap request scrubs the key, mirroring howblock_start_times/block_insert_timesare refreshed by each new claimant, so a marker orphaned by a missed completion cannot mis-tag a later request reusing the key.block_rq_completehonors a marker only when consistent with the matched issue context:marker_ts <= ictx->ts, no older than 1 s, and a write op (rwbs[0] == 'W'— a swap-tagged read is definitionally bogus). The marker is deleted on every completion path.is_swapbyte is appended to theflagscolumn in userspace via a unit-testableFlagMapper.append_swap_flag.Degradation
REQ_SWAP(< 4.19): the handler is compiled out via#ifdef;KernelProbeTrackerprobes for it withload_funcbefore attaching, so old kernels don't pay a per-request kprobe trap for a no-op.swaptag, nothing else affected.block_rq_*events, so they are invisible to the block stream with or without this change (documented).Docs
BLOCK_IO_EVENTS.mdgains a "Swap-origin tagging" section (mechanism + caveats); the flags field row andschema.pydescription mentionswap. Also aligned stale wording in the same doc with the code: rwbs sub-flags live in theflagscolumn (notoperation), the correlation key is a struct with separate dev/sector fields, and mainline tracepoints never exposedcmd_flags.Testing
SwapFlagTestsintests/test_flag_mapper.py(kernel-free, matching the file's pattern), including a test mirroring the_print_event_blockpipeline shape. Full suite: 136 tests pass (13 skipped, as before).disk_devt/blk_rq_trace_sectorformulas, map lifecycle) but cannot be exercised end-to-end here: that requires a swap-active host with root, e.g. a low-RAM VM understress-ng --vm, checkingswap-tagged write rows against/proc/vmstatpswpout— as the issue itself notes.🤖 Generated with Claude Code
https://claude.ai/code/session_0181MzEpBhHkug2Ftfg53G4L
Generated by Claude Code