Skip to content

ci(build): AUT-1064 provision NCCL 2.30 for source builds - #5112

Open
svcnemo-autobot wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:ci/provision-nccl-source-builds
Open

ci(build): AUT-1064 provision NCCL 2.30 for source builds#5112
svcnemo-autobot wants to merge 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:ci/provision-nccl-source-builds

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Background

TE 2.18 source-builds NCCL EP, but release and installation workflows used PyTorch images with incompatible NCCL headers/runtime.

What changed

  • Build workflow images with NCCL 2.30.7-1+cuda13.3 runtime and dev packages.
  • Use them for release-wheel and Installation Test builds.
  • Assert package versions, required headers/APIs, and the dynamically loaded NCCL library.

Details

docker/common/install_nccl.sh verifies nccl_device.h, EP declarations, loaded runtime ownership, and API version. docker/Dockerfile.nccl_ci derives from each workflow’s existing PyTorch base. NCCL EP remains enabled.

Tested

  • Full pre-commit, workflow YAML parsing, ShellCheck, bash -n, and git diff --check
  • Exact image reports NCCL API 23007 with required EP headers/declarations
  • Installation Test passed
  • Release build-wheel, validation, artifact download, and publish-wheel passed
  • CICD NeMo completed with 128 successful and 3 intentionally skipped jobs

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test 80769f2

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test a5a0295

@yaoyu-33 yaoyu-33 added area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work needs-review PR is ready for code review and waiting on a reviewer labels Jul 26, 2026
@svcnemo-autobot

Copy link
Copy Markdown
Collaborator Author

/ok to test a5a0295

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build Dependencies, packaging, images, and environment setup ci CI, automation, test queue, or workflow infrastructure work full-test-suite needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants