S06 memory validation triggers OOM with multiple stressapptest instances

[2012huynhdat-cmyk]

Auto-Tuner S06 causes OOM with concurrent stressapptest instances

System Info

• CPU: AMD Ryzen 7 5700X3D
• Zen generation: Zen 3
• Motherboard: ASUS ROG Strix B550-F Gaming
• BIOS version: 3404
• Kernel: 7.2.3-1-cachyos
• ryzen_smu version: 56.76.0
• CoreCycler version: fa42878
• Install method: from source
• RAM: 32 GB DDR4
• Distribution: CachyOS

Description

The Auto-Tuner reaches S06 memory validation, but stressapptest processes are repeatedly killed by the Linux OOM killer.

CoreCycler retries the affected tests as apparatus faults and eventually stops the validation after repeated failures.

The current stressapptest backend does not pass an explicit -M memory limit:

return [
    self.require_binary(),
    "-W",
    "-s",
    "86400",
]

During S06, multiple stressapptest instances run concurrently. On my 8-core Ryzen 7 5700X3D this eventually exhausts system memory.

The kernel confirms that the stressapptest processes are being killed because of OOM.

Expected Behavior

S06 should complete without exhausting system memory.

If multiple stressapptest instances are launched concurrently, I would expect the available memory budget to be divided between them, or for each instance to receive an appropriate memory limit.

Steps to Reproduce

1. Run the Auto-Tuner normally on a Ryzen 7 5700X3D with 32 GB RAM.
2. Allow all cores to complete tuning and reach S06 memory validation.
3. During S06, several stressapptest processes run concurrently.
4. The Linux OOM killer starts killing stressapptest processes.
5. CoreCycler reports apparatus faults and eventually stops the validation.

Relevant Output

Terminal output / error messages

Core 6: apparatus fault (Stress process killed externally (code -9) — possible OOM or system issue) — retrying the same step without a verdict (1/3)

Core 1: apparatus fault (Stress process killed externally (code -9) — possible OOM or system issue) — retrying the same step without a verdict (2/3)

Core 1: apparatus fault (Stress process killed externally (code -9) — possible OOM or system issue) — retrying the same step without a verdict (3/3)

Stress apparatus failed 4 times in a row (killed: Stress process killed externally (code -9) — possible OOM or system issue) — the environment cannot run this test, and repeating it would prove nothing. Stopping with offsets reverted to baseline; fix the cause (backend install, foreign load, permissions), then Resume.

Kernel log confirms actual OOM kills:

oom-kill:constraint=CONSTRAINT_NONE,... task=stressapptest
Out of memory: Killed process 2880 (stressapptest) ...
Out of memory: Killed process 2891 (stressapptest) ...
Out of memory: Killed process 2993 (stressapptest) ...
Out of memory: Killed process 3004 (stressapptest) ...

A manual memory test with an explicit 24 GB limit completes successfully:

$ stressapptest -s 300 -M 24576 -m 8 -W

Stats: Found 0 hardware incidents
Stats: Completed: 12687894.00M in 300.02s 42289.85MB/s, with 0 hardware incidents, 0 errors
Status: PASS - please verify no corrected errors

I also checked the kernel journal during this manual test and no MCE, EDAC, OOM, killed-process or segfault messages were reported.

corecycler doctor (always, for a backend or missing-tool problem)

Not included yet — I can provide the full `corecycler doctor` output if needed.

dmesg (if MCE/crash related)

No MCE was observed. The relevant kernel messages were OOM kills of stressapptest, shown above.

Additional Context

As a local workaround, I modified the stressapptest backend to add a 3072 MB limit per instance:

return [
    self.require_binary(),
    "-W",
    "-M",
    "3072",
    "-s",
    "86400",
]

With 8 concurrent instances this limits the requested memory to about 24 GB in total and leaves approximately 8 GB for the OS and other processes.

With this workaround, S06 completed successfully and the Auto-Tuner proceeded to validation stage 7.

The persisted tuner state after S06 showed:

status = validating
validation_stage = 7
validation_index = 0
validation_half = 2
validation_dirty = 0

The log also confirms the transition:

Validation stage 6: memory-load stress on all 8 cores simultaneously (300s, all offsets applied)
Validation cursor: stage=7 index=0 half=2 dirty=False requeue=[]
Validation stage 6: real-world soak — watching the kernel error stream for 1800s with no synthetic load.

There also appears to be a separate stage-number/display issue after S06 completes:

the persisted validation state is correctly validation_stage = 7
no stressapptest process is running anymore
the soak worker is active
but the GUI still displays:
Validating S6 (memory) — S6: 0/1

and the log says:

Validation stage 6: real-world soak

even though the validation cursor is already at stage 7.

Would it make sense for S06 to calculate a global memory budget and divide it between the concurrent stressapptest instances?

• Runs as root
• ryzen_smu module loaded
• Issue occurs consistently (not intermittent)