fix(recipes): MB-922 reduce DeepSeek VR200 memory pressure - #5086
fix(recipes): MB-922 reduce DeepSeek VR200 memory pressure#5086svcnemo-autobot wants to merge 3 commits into
Conversation
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
|
/ok to test a998cbd |
|
The VR200 GB200 → GB300 reassignment was introduced by |
@svcnemo-autobot lets keep the gb300 recipe and find another fix |
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
|
💬 Amended PR #5086 as requested. It now preserves the GB300-derived topology and reduces host memory by disabling 8 persistent dataloader subprocesses per rank for only the affected VR200 recipes. Formatting, Ruff, and 423 focused recipe tests pass. |
|
/ok to test 7124bd6 |
Background
MB-922 reports Linux/SLURM OOM kills before iteration 1 in the 256-GPU VR200 BF16 and NVFP4 DeepSeek-V3 jobs. Each rank inherited eight persistent dataloader workers, multiplying host processes and pinned-memory pressure per node.
What changed
dataset.num_workers = 0for only the affected VR200 recipes.Details
The PP/VPP/EP topology, micro-batch sizes, recompute modules, HybridEP domains, and precision settings remain unchanged. Removing dataloader subprocesses targets host RAM—the resource implicated by SLURM
oom_kill—without replacing the tuned GB300 model configuration.Tested
The exact 256-GPU VR200 jobs remain for CI hardware validation.