sanitizers: nightly red since 2026-09-02 — rj_waitcheck CFG dataflow does not converge on ROCm 10's gemm_NT_M256_N4096_K1024, so the case yields error against an expected warn

[vivekkhandelwal1]

sanitizers-nightly.yml has failed on its schedule since 2026-09-02. The gate job
gfx950 sanitizer regression fails because rj_waitcheck cannot complete its dataflow
analysis on one of the two f32 GEMM code objects the ROCm 10 base image ships, so that
kernel produces no verdict at all and the run aggregates to error against a baseline
that expects warn.

This is not the #450 mechanism: different sanitizer, different job, and #450's ConSan
cases pass here (see the control below).

Timeline

When (UTC)	Run	Trigger	Gate job
2026-09-01 12:24	33507543444	schedule	success — last green
2026-09-01 13:08	30dcc055 (#411)	merge	CI base image moved to ROCm 10
2026-09-02 12:26	33629901482	schedule	failure — first run on the new image
2026-09-03 10:22	33743980591	workflow_dispatch	failure — byte-identical signature

The last green ran 44 minutes before the merge, which is as tight a bracket as a
nightly can give: nothing else landed in between.

Note

The run list is misleading if you read only the conclusion column. Run
33743980591 shows a run-level conclusion of cancelled, because the
non-gating gfx950 workload survey job was cancelled. The gate job
gfx950 sanitizer regression concluded failure in both 33629901482 and
33743980591.

#411 swapped the Dockerfile.ci-gpu base:

- rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0@sha256:4449f856…
+ rocm/pytorch:rocm10.0_ubuntu26.04_py3.14_pytorch_release_2.13.0@sha256:3174cb70…

Mechanism

scripts/sanitizers/prepare_gemm_isa.py extracts the f32 GEMM code objects from the
image's own rocBLAS/Tensile bundle, so the gate scans whatever the image ships. Those
objects changed with the image:

Case	Old object	New object
gemm_NT_M256_N4096_K1024	5bd40b78…	57c5d8ef…
gemm_TT_M64_N64_K1280	7ea836fe…	aeb46fde…

On the new gemm_NT_M256_N4096_K1024, rj_waitcheck reports:

sol_126578.hsaco: waitcheck analysis failed for gfx950[0]:
waitcheck CFG dataflow did not converge at .text+0x24d7070
(in:loadcnt,uncertain-order;out:loadcnt,uncertain-order)

No analysis summary is produced, so the kernel result is waitcheck_diagnostics_error,
returncode 2, verdict error. That aggregates to an overall error, and
compare_verdict_baselines.py rejects it against the committed expectation in
recipes/sanitizers/fixtures/expected/verdict_baselines.json:

"waitcheck_gemm": {
  "overall_verdict": "warn",
  "execution_status": "complete",
  "checks": { "waitcheck": "warn" },
  "finding_shape": { "waitcheck": "missing s_waitcnt" }
}

Same offset both nights, so this is deterministic and content-driven, not flaky.

Two exclusions, both with positive evidence

These are what make the diagnosis solid, so they go before the discussion rather than
after it.

1. The analyser is not the variable. The rj_waitcheck binary is byte-identical
across all three runs — sha256 a70945fb1135a436c13c83ed… — even though the RocJITsu
bundle commit differs nightly. A nightly-changing bundle that nevertheless ships the same
binary is exactly the case where you would otherwise suspect the tool; it is excluded by
measurement, not by assumption.

2. Device access, capabilities and instrumentation are all fine. In the same failing
job, the ConSan cases returned pass and fail exactly as baselined
(consan_clean → pass, consan_racy → fail) while logging
target=gfx950 arch=cdna4. ConSan is the dynamic sanitizer: if the device were
unreachable, the arch were misdetected, or instrumentation were missing, it would fail
first and hardest. It didn't.

Together those two leave the code object as the only thing that changed.

Hardware is not a factor either way: rj_waitcheck is a static, CPU-side analyser
that never touches the GPU.

What the defect is — please choose, because there are two defensible readings

I do not think this should be decided by whoever fixes it first, so both are stated:

(a) rj_waitcheck has a convergence bug. The CFG dataflow fails to reach a fixed
point on a code object it should be able to analyse. The uncertain-order state on both
sides of the reported point suggests the analysis cannot order loadcnt events across
some construct the ROCm 10 Tensile kernels now emit, and gives up rather than widening to
a conservative answer. Under this reading the baseline is correct and the tool needs
fixing.

(b) The committed expectation is stale. The ROCm 10 objects are genuinely different
kernels; warn / "missing s_waitcnt" described the ROCm 7.2 object, and the fixture needs
re-blessing against what the new image ships.

Important

Re-blessing to error would be the wrong fix. error does not mean "a different
finding"; it means no signal was produced. Accepting it as the expected value
converts a real coverage gap into a green check — the same anti-pattern already
documented in #450, where a load-only driver paired with consan_policy: strict can
never return a verdict. A green nightly would then assert that this kernel is checked
when nothing checks it.

So: settle (a) before any re-bless. If rj_waitcheck is supposed to analyse this
object, the fixture is right and the tool is wrong, and re-blessing destroys the evidence.
If the analyser genuinely cannot handle it, that is a known limitation to record
explicitly — as a skip or an expected-limitation state that is visibly not coverage —
rather than by writing error into the expected column.

If the RocJITsu side turns out to be implicated, ROCm/rocm-systems#10966
may be worth a look. Recorded as an observation only — I am not asserting a shared
cause.

Impact

• The nightly sanitizer gate has been red for two consecutive nights.
• The gemm_NT_M256_N4096_K1024 case now contributes no waitcheck signal at all, so
  that kernel is unchecked regardless of how the gate is scored.

Reproduce

Actions -> sanitizers-nightly -> Run workflow

It reproduces on demand — 33743980591 was a manual dispatch and matched the scheduled
failure byte for byte.