sanitizers: daily-consan-tiny and daily-consan-gemm pair a load-only driver with consan_policy: strict, so they can never return a verdict

[vivekkhandelwal1]

Summary

daily-consan-tiny.yaml and daily-consan-gemm.yaml drive ConSan through consan_load.hip, which loads and instruments a code object but deliberately never dispatches it. Both recipes also set consan_policy: strict, and strict sets RJ_CONSAN_MOI_REQUIRE_RECORDS, which requires visible dynamic records. A driver that never dispatches cannot produce records, so these two recipes fail closed every time. They cannot return a verdict under their current configuration.

ConSan is behaving correctly here and says so in as many words:

[rocjitsu-dbi-hooks] RJ_CONSAN_MOI_REQUIRE_RECORDS requested, but 0 auto MOI report
buffer(s) contained zero visible records and no kernel dispatch packet was observed

The recipes surface that as overall_verdict=error, execution_status=error, reason combined_hook_exit_86, with zero findings. This is a recipe-configuration defect, not a rocjitsu bug.

Why it happens

• consan.py:662 sets RJ_CONSAN_POLICY=strict from consan_policy: strict.
• consan_load.hip:4 states its own contract: "No kernel dispatch (production StreamK GEMM kernels need hipBLASLt to launch)". It calls hipModuleLoad and hipModuleUnload and nothing else.
• No dispatch means no records, and RJ_CONSAN_MOI_REQUIRE_RECORDS then rejects the run.

The contrast with the third caller-supplied-object recipe is what localises this. daily-consan-lds-dispatch.yaml also supplies its own code object and also runs consan_policy: strict, but its driver lds_dispatch.hip:54 calls hipModuleLaunchKernel, so it produces records and returns a real pass. The discriminator is therefore whether the driver dispatches, not whether the object is caller-supplied.

This repo already knows the rule and applies it correctly elsewhere. harvest_code_objects.py:933-938 emits consan_policy: lenient for its own load-mode recipes and explains why in a comment that describes this exact failure:

strict sets RJ_CONSAN_MOI_REQUIRE_RECORDS, which demands visible dynamic records. The loader runs in load mode: it loads and instruments the object but never dispatches it, so there is no dispatch packet and no records, and strict fails closed with combined_hook_exit_86 no matter how healthy the run was.

So the fix pattern is established in the tree; the two daily recipes just do not follow it.

Probable provenance: #359's action was "flip consan_policy: lenient → strict in the affected ConSan recipes", and strict landed on these recipes in 1663ad7 (2026-08-11, #362). #359's acceptance criterion was that a strict run returns a real verdict "not combined_hook_timeout" — which is literally satisfied, because these now fail with exit 86 instead, so the regression did not trip the check that was watching for it.

It is reproducible, not flaky, and has been failing for some time

Fresh run, all nine daily-* recipes on one node:

recipe	driver	dispatches?	verdict	reason
daily-consan-tiny	consan_tiny_load	no	error	combined_hook_exit_86
daily-consan-gemm	consan_gemm_load	no	error	combined_hook_exit_86
daily-consan-lds-dispatch	lds_dispatch	yes	pass	—
daily-consan-clean	consan_lds_race	yes	pass	—
daily-consan-racy	consan_lds_race_2wave	yes	fail	—

The six reports committed under recipes/sanitizers/survey/reports/ show the same thing, so this is not new: every ConSan case there is a strict-policy rejection, two as combined_hook_exit_86 and one as consan_strict_load_rejection (exit 92, the recognised sibling of the same policy check), while all three Waitcheck cases over the same objects complete.

It went unnoticed because the recipes report error rather than failing loudly, and both are informational/non-gating, so nothing downstream treats an error verdict as a broken run.

Impact

Two of the nine daily ConSan/Waitcheck recipes produce no sanitizer signal at all — zero findings, no verdict, no coverage numbers. Any coverage or "ConSan runs clean on these objects" claim derived from them is empty rather than positive: the tool never got far enough to have an opinion. That matters most for daily-consan-gemm, which is the only daily case pointed at a real Tensile GEMM object, so the heaviest and most representative object in the suite is the one contributing nothing.

Note also that daily-consan-tiny's object genuinely has no instrumentable sites (supported_lds_sites=0, and the recipe description says so), so even once the policy is fixed that case should be expected to return a trivially clean verdict rather than useful coverage.

Environment

Slurm node cv350-rck-g03-c17-08, partition meta64, gfx950, ROCm 7.0.2.2. RocJITsu prebuilt bundle f92da4cb3c5b3612db3752a36f4f3d0d3e9ff768 (rocm-systems Actions run 33641388794). aorta at ba73dab. Reproduced by running the nine daily-* recipes exactly as .github/workflows/sanitizers-nightly.yml does.

Suggested remedies

Not prescribing one.

1. Set consan_policy: lenient on the two load-only recipes, matching what harvest_code_objects.py already does for load-mode recipes. Smallest change, and it makes these cases verify what they can actually verify: static instrumentation coverage.
2. Give the load-only recipes a dispatching driver, so strict becomes meaningful. Real work for consan_gemm_load — the comment in consan_load.hip notes StreamK GEMM kernels need hipBLASLt to launch, which is why it is load-only in the first place.
3. Reject the combination at recipe-validation time, so consan_policy: strict plus a driver known not to dispatch is a loud configuration error rather than a runtime error verdict. This is the one that stops it recurring; the other two fix today's instance.

I would do 1 now and 3 shortly after. 1 restores signal immediately and follows a pattern the repo already committed to, and 3 is what would have caught the #359 flip at the point it was made instead of leaving two recipes silently mute for weeks. 2 is worth having eventually but should not gate the others.

Discovery context

Found while generating a labelled corpus of real sanitizer runs for RL post-training, which meant running every daily-* recipe and checking each verdict against fixtures/expected/verdict_baselines.json. The three baseline-gated cases all agreed with their baselines; these two are among the informational cases, which have no baseline entry, and that is why an error there is not currently caught by anything.