Skip to content

fix(recipes): MB-922 reduce DeepSeek VR200 memory pressure - #5086

Open
svcnemo-autobot wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/deepseek-v3-vr200-oom
Open

fix(recipes): MB-922 reduce DeepSeek VR200 memory pressure#5086
svcnemo-autobot wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/deepseek-v3-vr200-oom

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Background

MB-922 reports Linux/SLURM OOM kills before iteration 1 in the 256-GPU VR200 BF16 and NVFP4 DeepSeek-V3 jobs. Each rank inherited eight persistent dataloader workers, multiplying host processes and pinned-memory pressure per node.

What changed

  • Keep both recipes derived from their GB300 configurations.
  • Set dataset.num_workers = 0 for only the affected VR200 recipes.
  • Add regression tests for the GB300 topology and worker limit.

Details

The PP/VPP/EP topology, micro-batch sizes, recompute modules, HybridEP domains, and precision settings remain unchanged. Removing dataloader subprocesses targets host RAM—the resource implicated by SLURM oom_kill—without replacing the tuned GB300 model configuration.

Tested

  • Ruff format check: passed.
  • Ruff lint: passed.
  • Focused recipe suites: 423 passed.

The exact 256-GPU VR200 jobs remain for CI hardware validation.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test a998cbd

@svcnemo-autobot svcnemo-autobot added needs-more-tests Requires additional L0 and L1 test coverage before merge labels Jul 24, 2026
@yaoyu-33 yaoyu-33 added area:perf Performance optimizations and benchmarking bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Jul 24, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

The VR200 GB200 → GB300 reassignment was introduced by bf07e5678cb96433467d0d495fc898de4a6849c2 (vr200 cfgs match w/ gb300, PR #4574). It changed the DeepSeek-V3 VR200 BF16 and NVFP4 aliases from their GB200 recipes to GB300. This PR restores the GB200-derived layouts for those two affected recipes.

@ko3n1g

ko3n1g commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

The VR200 GB200 → GB300 reassignment was introduced by bf07e5678cb96433467d0d495fc898de4a6849c2 (vr200 cfgs match w/ gb300, PR #4574). It changed the DeepSeek-V3 VR200 BF16 and NVFP4 aliases from their GB200 recipes to GB300. This PR restores the GB200-derived layouts for those two affected recipes.

@svcnemo-autobot lets keep the gb300 recipe and find another fix

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

💬 Amended PR #5086 as requested. It now preserves the GB300-derived topology and reduces host memory by disabling 8 persistent dataloader subprocesses per rank for only the affected VR200 recipes. Formatting, Ruff, and 423 focused recipe tests pass.

@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 7124bd6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:perf Performance optimizations and benchmarking bug Something isn't working needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants