Skip to content

[DO NOT MERGE] wheels: build CUDA 13 wheels with latest CTK (13.3.0)#487

Open
jameslamb wants to merge 8 commits into
rapidsai:mainfrom
jameslamb:wheels-latest-ctk
Open

[DO NOT MERGE] wheels: build CUDA 13 wheels with latest CTK (13.3.0)#487
jameslamb wants to merge 8 commits into
rapidsai:mainfrom
jameslamb:wheels-latest-ctk

Conversation

@jameslamb

@jameslamb jameslamb commented Jul 8, 2026

Copy link
Copy Markdown
Member

Contributes to rapidsai/build-planning#268

Part of a RAPIDS-wide initiative to build wheels using v13.3.0 of the CUDA Toolkit, and to generally get RAPIDS back to the pattern of always building against its latest supported CTK.

Notes for Reviewers

The workflow-branch changes here will be reverted before merging.

@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jameslamb jameslamb added breaking Introduces a breaking change improvement Improves an existing functionality labels Jul 8, 2026
@jameslamb jameslamb added non-breaking Introduces a non-breaking change and removed breaking Introduces a breaking change labels Jul 8, 2026
Comment thread dependencies.yaml Outdated
@jameslamb

jameslamb commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

Builds (conda and wheels) are failing like this:

CMakeFiles/wholegraph.dir/src/wholegraph_ops/weighted_sample_without_replacement_impl_mapped.cu.o
 │ │ $BUILD_PREFIX/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/atomic(218): error: static assertion failed with "std::atomic requires a trivially copyable type"
 │ │         static_assert(__is_trivially_copyable(_Tp),
 │ │         ^
 │ │           detected during instantiation of class "std::atomic<_Tp> [with _Tp=std::shared_ptr<raft::resource::resource_factory>]" at line 95 of $PREFIX/include/raft/core/resource/resource_types.hpp
 │ │ $BUILD_PREFIX/lib/gcc/x86_64-conda-linux-gnu/14.3.0/include/c++/atomic(218): error: static assertion failed with "std::atomic requires a trivially copyable type"
 │ │         static_assert(__is_trivially_copyable(_Tp),
 │ │         ^
 │ │           detected during instantiation of class "std::atomic<_Tp> [with _Tp=std::shared_ptr<raft::resource::resource>]" at line 96 of $PREFIX/include/raft/core/resource/resource_types.hpp
 │ │ 2 errors detected in the compilation of "$SRC_DIR/cpp/src/wholegraph_ops/weighted_sample_without_replacement_impl_mapped.cu".

(build link)

Believe this was broken by NVIDIA/raft#3052

We'll need new cugraph packages published to fix that.

update: merged a small PR to cugraph (rapidsai/cugraph#5577), which triggered a new build (link), once that completes this issue should be resolved.

@jameslamb

Copy link
Copy Markdown
Member Author

/ok to test

Comment thread .github/workflows/pr.yaml
- wheel-tests-pylibwholegraph-nightly
- wheel-build-cugraph-pyg
- wheel-tests-cugraph-pyg
- wheel-tests-cugraph-pyg-nightly

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO before merging: remove these

Just added to test that the nightly matrices would work too.

@jameslamb

Copy link
Copy Markdown
Member Author

/ok to test

wheel-build-cugraph-pyg:
secrets: inherit # zizmor: ignore[secrets-inherit]
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@main
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@newest-ctk

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just picking a random place on the diff so this conversation can be threaded and eventually resolved... still seeing the issues from #487 (comment), so maybe the entire chain of packages picking up new builds for the RAFT ABI break didn't make it up yet.

Just pushed another commit, trying again. If that fails, I'll go look more closely.

@jameslamb jameslamb changed the title WIP: wheels: build CUDA 13 wheels with latest CTK (13.3.0) [DO NOT MERGE] wheels: build CUDA 13 wheels with latest CTK (13.3.0) Jul 8, 2026
@jameslamb jameslamb added the DO NOT MERGE Hold off on merging; see PR for details label Jul 8, 2026
@jameslamb jameslamb marked this pull request as ready for review July 8, 2026 21:09
@jameslamb jameslamb requested review from a team as code owners July 8, 2026 21:09
@jameslamb jameslamb requested a review from KyleFromNVIDIA July 8, 2026 21:09
@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR is part of a RAPIDS-wide initiative to build wheels against CUDA 13.3.0 (CTK 13.3). It switches wheel-build jobs to a temporary @newest-ctk branch of rapidsai/shared-workflows, adds nightly test matrix jobs for CUDA 13.x, consolidates cuda-toolkit pins in dependencies.yaml into wildcard ranges, and removes the per-CUDA-version torch availability guards from CI scripts (now all supported CUDA versions are assumed to have torch wheels).

  • Wheel-build shared-workflow refs temporarily point to @newest-ctk; per the PR description, these will be reverted to @main before merging.
  • download-torch-wheels.sh loses its skip guard, and the test scripts remove their torch_downloaded conditional — tests now hard-fail if a torch wheel is unavailable, replacing the previous graceful-skip path.
  • dependencies.yaml gains a cuda: \"13.*\" catchall using torch==2.13.0+cu132 (covering CUDA 13.2 and 13.3), and a cuda: \"12.*\" wildcard range replaces the previously explicit 12.2/12.5 pins.

Confidence Score: 4/5

Infrastructure-only CI/dependency change; no production code is modified. The main risk is that adding nightly jobs as required PR checks could block unrelated PRs if CUDA 13 tests are not yet stable.

All changes are CI scripts and dependency declarations. The unconditional torch download/test path is a deliberate tightening of assumptions that the author has justified. One comment is incomplete, and the nightly jobs becoming required checks warrants a second look before final merge.

.github/workflows/pr.yaml — the two new nightly jobs added to pr-builder deserve confirmation that they won't gate unrelated PRs on potentially unstable CUDA 13.3 runners. dependencies.yaml — the incomplete comment around the cuda: "13.*" cuda-toolkit entry.

Important Files Changed

Filename Overview
.github/workflows/build.yaml Switches wheel-build jobs to @newest-ctk branch of shared-workflows; workflow-branch changes will be reverted before merging per PR description.
.github/workflows/pr.yaml Switches wheel-build jobs to @newest-ctk and adds two new nightly test jobs (wheel-tests-pylibwholegraph-nightly, wheel-tests-cugraph-pyg-nightly) as required checks in pr-builder.
ci/download-torch-wheels.sh Removes the CUDA-version guard that previously skipped torch download for unsupported CTK versions; now always attempts download, assuming all supported CUDA versions have torch wheels.
ci/test_wheel_cugraph-pyg.sh Removes torch_downloaded conditional guard; torch install and tests now run unconditionally, reflecting the assumption that all CI CUDA versions have torch wheel support.
ci/test_wheel_pylibwholegraph.sh Same pattern as test_wheel_cugraph-pyg.sh: removes torch_downloaded conditional and unconditionally asserts torch is installed with CUDA support.
dependencies.yaml Drops pinned cuda-toolkit entries for 12.2/12.5, replaces with a cuda: "12." wildcard range; adds cuda: "13." catchall using cu132 torch wheels for CUDA 13.2+; has one incomplete comment.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CI Trigger] --> B[wheel-build jobs\n@newest-ctk branch]
    B --> C[wheel-build-libwholegraph]
    B --> D[wheel-build-cugraph-pyg]
    C --> E[wheel-build-pylibwholegraph]

    E --> F[wheel-tests-pylibwholegraph\nstable matrix]
    E --> G[wheel-tests-pylibwholegraph-nightly\nnightly matrix - NEW]
    E --> H[wheel-tests-cugraph-pyg\nstable matrix]
    D --> H
    E --> I[wheel-tests-cugraph-pyg-nightly\nnightly matrix - NEW]
    D --> I

    F --> J[pr-builder\nrequired gate]
    G --> J
    H --> J
    I --> J
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[CI Trigger] --> B[wheel-build jobs\n@newest-ctk branch]
    B --> C[wheel-build-libwholegraph]
    B --> D[wheel-build-cugraph-pyg]
    C --> E[wheel-build-pylibwholegraph]

    E --> F[wheel-tests-pylibwholegraph\nstable matrix]
    E --> G[wheel-tests-pylibwholegraph-nightly\nnightly matrix - NEW]
    E --> H[wheel-tests-cugraph-pyg\nstable matrix]
    D --> H
    E --> I[wheel-tests-cugraph-pyg-nightly\nnightly matrix - NEW]
    D --> I

    F --> J[pr-builder\nrequired gate]
    G --> J
    H --> J
    I --> J
Loading

Reviews (1): Last reviewed commit: "empty commit to re-trigger CI" | Re-trigger Greptile

Comment thread dependencies.yaml
Comment on lines +270 to +272
# 'torch' uses '==' pins for its cuda-toolkit dependency and only supports certain versions
#
# ... allowing this to float allows us

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The comment is cut off mid-sentence, leaving the rationale unexplained.

Suggested change
# 'torch' uses '==' pins for its cuda-toolkit dependency and only supports certain versions
#
# ... allowing this to float allows us
# 'torch' uses '==' pins for its cuda-toolkit dependency and only supports certain versions,
# so an exact pin here would conflict. Allowing this to float lets the solver pick a
# compatible CTK version (13.1+) without fighting torch's own constraints.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment thread .github/workflows/pr.yaml
Comment on lines 23 to 30
- wheel-build-libwholegraph
- wheel-build-pylibwholegraph
- wheel-tests-pylibwholegraph
- wheel-tests-pylibwholegraph-nightly
- wheel-build-cugraph-pyg
- wheel-tests-cugraph-pyg
- wheel-tests-cugraph-pyg-nightly
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@main

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Nightly jobs gating PR merges

wheel-tests-pylibwholegraph-nightly and wheel-tests-cugraph-pyg-nightly are now required checks in pr-builder. If nightly matrix tests against CUDA 13.3 are flaky or not yet stable, every PR would be blocked. This looks intentional here, but confirm this remains the right gating strategy when rebasing onto the final shape of the newest-ctk branch changes.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DO NOT MERGE Hold off on merging; see PR for details improvement Improves an existing functionality non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant