Move driver and nvrtc cython and internal layers to new generator by mdboom · Pull Request #1972 · NVIDIA/cuda-python

mdboom · 2026-04-24T13:46:37Z

This is a continuation of the work in #1900. Now adds driver to the mix and both nvrtc and driver are generated from the "real" new generator.

copy-pr-bot · 2026-04-24T13:46:40Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

mdboom · 2026-04-24T13:47:21Z

/ok to test

mdboom · 2026-04-24T14:59:54Z

/ok to test

mdboom · 2026-04-24T17:37:15Z

/ok to test

mdboom · 2026-04-24T17:52:22Z

/ok to test

mdboom · 2026-04-24T19:05:25Z

/ok to test

mdboom · 2026-04-24T20:48:19Z

/ok to test

mdboom · 2026-04-24T21:30:36Z

/ok to test

mdboom · 2026-04-25T14:21:56Z

/ok to test

mdboom · 2026-04-25T15:17:05Z

/ok to test

mdboom · 2026-04-25T15:17:34Z

/ok to test

mdboom · 2026-04-29T13:38:07Z

/ok to test

leofang · 2026-06-10T19:02:01Z

It turns out the exception signature needs to match exactly when declaring function pointer types (which cuda-core does in one spot).

I assume that this also helps us avoid any potential (Cython) ABI breakage?

leofang · 2026-06-10T22:13:31Z

I wish we have the ABI test running in the CI 😛 Eyeball'd as much as I can. Thanks, Mike!

The CI flakiness should be fixed in #2195. I retried a few times here to make it green.

leofang

Re-reviewing the symbol-loading rework end-to-end against main. Two file-anchored concerns below, plus one branch-state issue:

Missing CUDA 13.3 symbols. The following are declared and loaded in _bindings/cydriver.pyx.in on main (all at cudaVersion=13030) but absent from every new file in this PR (driver_linux.pyx, driver_windows.pyx, _internal/driver.pxd, cydriver.pyx):

cuLogicalEndpoint{Create,Destroy,AddDevice,BindAddr,BindMem,Unbind,Export,Import,Query,GetLimits,IdRelease,IdReserve} — 12 functions
cuStreamBeginRecaptureToGraph — 1 function

Looks like the branch was regenerated before these landed on main. Calling any of them post-merge would be a hard breakage, so this needs a rebase + regen before going in. Not a design fix.

Cancelling my prior approval until that and the inline comments are addressed.

-- Leo's bot

mdboom · 2026-06-11T13:26:07Z

It turns out the exception signature needs to match exactly when declaring function pointer types (which cuda-core does in one spot).

I assume that this also helps us avoid any potential (Cython) ABI breakage?

It does. It's overly strict (it wouldn't have been a breakage in this specific case), but at least if we /really/ broke it we know Cython would tell us.

mdboom · 2026-06-11T14:28:03Z

@leofang: I have updated this to include the ptds version numbers where appropriate and also added ptds testing to a couple of the configurations in CI.

Co-authored-by: Leo Fang <leof@nvidia.com>

leofang · 2026-06-11T21:48:38Z

@mdboom this regression is not yet addressed?

Missing CUDA 13.3 symbols. The following are declared and loaded in _bindings/cydriver.pyx.in on main (all at cudaVersion=13030) but absent from every new file in this PR (driver_linux.pyx, driver_windows.pyx, _internal/driver.pxd, cydriver.pyx):

cuLogicalEndpoint{Create,Destroy,AddDevice,BindAddr,BindMem,Unbind,Export,Import,Query,GetLimits,IdRelease,IdReserve} — 12 functions

cuStreamBeginRecaptureToGraph — 1 function

mdboom · 2026-06-12T12:25:21Z

@mdboom this regression is not yet addressed?

Missing CUDA 13.3 symbols. The following are declared and loaded in _bindings/cydriver.pyx.in on main (all at cudaVersion=13030) but absent from every new file in this PR (driver_linux.pyx, driver_windows.pyx, _internal/driver.pxd, cydriver.pyx):

cuLogicalEndpoint{Create,Destroy,AddDevice,BindAddr,BindMem,Unbind,Export,Import,Query,GetLimits,IdRelease,IdReserve} — 12 functions

cuStreamBeginRecaptureToGraph — 1 function

Sorry, I didn't see this comment yesterday. These have been added now.

github-actions · 2026-06-13T00:48:24Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

* Move driver and nvrtc cython and internal layers to new generator (#1972) * Move driver and nvrtc cython and internal layers to new generator * Fix Cython interop tests * Handle headers differently * Fix compilation * Make const match * Fix compilation again * Attempt to fix Windows build * Fix Windows again * Fix not found error code * Look up pointers in a different way * Get first version right * Get first appearing versions correct * Fix weird special case * Fix exception signature to match what was there before * Also fix exception declaration in nvrtc * Fix PTDS and add PTDS testing * Update .gitignore Co-authored-by: Leo Fang <leof@nvidia.com> --------- Co-authored-by: Leo Fang <leof@nvidia.com> * Fix up generated code * Correctly set CUDA_VERSION --------- Co-authored-by: Leo Fang <leof@nvidia.com>

leofang · 2026-07-01T02:40:39Z

    - { ARCH: 'amd64', PY_VER: '3.11',  CUDA_VER: '13.0.2', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
    - { ARCH: 'amd64', PY_VER: '3.11',  CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
-    - { ARCH: 'amd64', PY_VER: '3.12',  CUDA_VER: '12.9.1', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
+    - { ARCH: 'amd64', PY_VER: '3.12',  CUDA_VER: '12.9.1', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest', ENV: { CUDA_PYTHON_PER_THREAD_DEFAULT_STREAM: '1' } }


Turns out that this is a typo... 😞 It should have been CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM. I'll fix it...

* CI: add nightly-cuda-core and nightly-numba-cuda-mlir modes nightly-cuda-core: test the released cuda-core from PyPI against main-built pathfinder and cuda-bindings, catching the "core released × bindings main" gap documented in issue #1955. Runs on linux-64 (a100) and win-64 (a100 MCDM). nightly-numba-cuda-mlir: MLIR-backend companion to nightly-numba-cuda. Installs main pathfinder+bindings+core plus numba-cuda-mlir from PyPI, runs numba-cuda-mlir's own test suite from the matching git tag. Linux amd64/arm64 x CUDA 12.9.1 / 13.3.0. Both modes fetch the released version's tests from git tags because the respective wheels do not ship test_*.py files. Includes tag-not-found fallback (log warning + exit 0) to avoid red-lining the nightly on a freshly-cut PyPI release that hasn't been pushed to git yet. * ci/test-matrix.yml: fix CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM typo The two ENV overrides intended to exercise the per-thread default stream code path were misspelled (missing the CUDA_ segment), so the env var was silently ignored and the PTDS coverage added in #1972 had no effect. Rename to the correct CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM. Refs #971. * cuda_pathfinder: pin nvshmem to <3.7 (was previously excluding only 3.7.0) nvidia-nvshmem-cu{12,13} 3.7.x breaks the main branch, not only 3.7.0. Widen the exclusion from an exact-version bump to <3.7 so 3.7.x and above are avoided until we can move forward. * nightly-numba-cuda-mlir: swap arm64 for win-64 coverage, use rtxpro6000 Drop the linux-aarch64 rows and instead add win-64 coverage with the same CUDA 12.9.1 / 13.3.0 pair. Switch all four rows from GPU l4 to rtxpro6000. Windows rows use DRIVER_MODE MCDM, matching the existing rtxpro6000 CUDA 13.3.0 patterns. * Temporarily add push trigger to ci-nightly.yml for testing Remove before merging. * CI: switch nightly-{cuda-core,numba-cuda-mlir} to actions/checkout for tests The initial approach used git inside the ubuntu:24.04 container to fetch the released version's test suite, but git is not installed on that container (install_unix_deps only pulls in jq/wget/g++/etc.) and its absence made the run steps silently skip via the tag-not-fetchable fallback. On Windows, git archive of just the cuda_core subtree also hit a dangling-symlink extraction failure (cuda_core/.git_archival.txt). Refactor to: - run-tests: just install wheels and expose the resolved release version (CUDA_CORE_RELEASED_VER / NUMBA_CUDA_MLIR_VER) and cuda-core test-group name via GITHUB_ENV. No more git operations. - test-wheel-{linux,windows}.yml: add an actions/checkout step per mode that pulls the matching release tag into a subdirectory (cuda-core-released / numba-cuda-mlir-released), then the follow-up test step installs that tag's test dep-group and runs pytest. For numba-cuda-mlir also pass --ignore=tests/benchmarks --ignore=tests/doc_examples to pytest: those directories import the `numba` package at module top and would fail collection, which is cuSIMT's expected behavior (see NVIDIA/numba-cuda-mlir#136 — cuSIMT intentionally does not depend on numba). * CI: pin numpy<2.5 (mlir) and pytest<9.1 (cuda-core released tests) Two nightly failure fixups after the first green iteration: nightly-numba-cuda-mlir: numba-cuda-mlir 0.4.0 has an inverted guard that registers an overload of np.row_stack on NumPy 2.x, and NumPy 2.5 removed that name entirely, so test collection fails with "AttributeError: module 'numpy' has no attribute 'row_stack'". Cap numpy to <2.5. See NVIDIA/numba-cuda-mlir#154. nightly-cuda-core: released cuda-core v1.0.1's test suite uses a parametrize argvalues pattern that pytest 9.1 rejects ("in parametrize the number of names (1)... must be equal to the number of values (3)"). The main-side fix was #2212 but it has not shipped in a cuda-core release yet. Cap pytest to <9.1 for the released-cuda-core test run only. * CI: deselect known pre-existing failures in nightly-cuda-core and nightly-numba-cuda-mlir Applied only in the affected nightly-* pytest invocations; the released source trees under test are unmodified. nightly-numba-cuda-mlir (all 10 tests deselected are from cuSIMT): * CudaArraySetting::{test_no_sync_default_stream, test_no_sync_supplied_stream, test_sync} TestCudaArrayInterface::{test_consume_no_sync, test_consume_sync, test_launch_no_sync, test_launch_sync, test_launch_sync_two_streams, test_fortran_contiguous} Serial-pytest contamination of numba_cuda_mlir.cuda.cudadrv from an xfailed test in test_nrt_comprehensive.py. Upstream CI runs with `pytest -n auto --dist loadscope`, which isolates the offending side effect in a separate xdist worker; our nightly runs serially and hits the pollution. See NVIDIA/numba-cuda-mlir#135. * TestLinkerDumpAssembly::test_nvjitlink_jit_with_linkable_code_lto_dump_assembly_warn Subprocess-invokes `cuobjdump`, which isn't on PATH in the base ubuntu:24.04 container. Filed as an upstream skip-guard bug. nightly-cuda-core (3 tests deselected are pre-existing v1.0.1 issues): * test_enum_coverage.py::test_wrapper_covers_all_binding_members[NvlinkVersion] Expected drift: main cuda-bindings adds NvlinkVersion.VERSION_6_0 which v1.0.1's wrapper mapping predates. This mode intentionally pairs released core with main bindings, so this coverage-style test will stay red here until a cuda-core release catches up. * test_rlcompleter_patch.py::test_opt_out_env_var_disables_patch_even_when_interactive Environment-dependent test: expects rlcompleter to crash without the tab-completion patch, but on Windows MCDM the pre-patch behavior is clean. Passes on Linux, fails on Windows MCDM. * test_memory.py::test_non_managed_resources_report_not_managed[pinned] Same underlying "Failed to allocate memory from pool" error that v1.0.1 already xfails in the sibling test_pinned_memory_resource_initialization (TODO(#9999)). cuda-python main has since fixed the parametrized case to route through _allocate_pinned_buffer_or_xfail(), but that fix hasn't shipped in a cuda-core release yet. * CI: tighten deselects to per-platform failing sets Previously applied the same list on both Linux and Windows workflows, which over-deselected — some tests only fail on one platform because the underlying issues (serial-pytest test-order in mlir, MCDM-only behavior in cuda-core) are platform-specific. Now: nightly-numba-cuda-mlir linux-64: TestCudaArrayInterface::{test_consume_no_sync, test_consume_sync, test_launch_no_sync, test_launch_sync, test_launch_sync_two_streams, test_fortran_contiguous} + TestLinkerDumpAssembly::test_nvjitlink_jit_with_linkable_code_lto_dump_assembly_warn. win-64: CudaArraySetting::{test_no_sync_default_stream, test_no_sync_supplied_stream, test_sync} + TestCudaArrayInterface::test_fortran_contiguous. Test-order contamination in numba-cuda-mlir#135 surfaces different tests depending on collection order (linux-64 vs win-64 exercise different subsets), so the per-platform lists differ. cuobjdump-based TestLinkerDumpAssembly only fires on Linux because the ubuntu:24.04 container's PATH lacks cuobjdump; Windows runners ship it with the local CTK. nightly-cuda-core linux-64: test_enum_coverage.py::test_wrapper_covers_all_binding_members[NvlinkVersion]. win-64: NvlinkVersion (same as Linux) + test_rlcompleter_patch.py::test_opt_out_env_var_disables_patch_even_when_interactive + test_memory.py::test_non_managed_resources_report_not_managed[pinned]. rlcompleter and pinned mempool tests only fail on Windows MCDM. NvlinkVersion fails on both (expected drift for the mode). * CI: version-gate the nightly-mode deselects so they auto-clean Each deselect is now wrapped in a bash conditional keyed on the installed release version. When a newer numba-cuda-mlir or cuda-core release ships with the referenced fix, the nightly picks it up automatically, the guard evaluates false, and the deselect drops — so the tests run against the new release. If they still fail we hear about it loudly rather than silently masking a regression. Current guards: - numba-cuda-mlir #135 tests + cuobjdump TestLinkerDumpAssembly: applied when installed numba-cuda-mlir version <= 0.4.0. - cuda-core NvlinkVersion / rlcompleter opt-out / pinned mempool: applied when installed cuda-core version <= 1.0.1. Structure keeps one conditional block per (mode, platform) with a comment above each deselect explaining the tracking issue. * CI: broaden mlir deselect list to full #135 union across platforms The previous per-platform-tight lists were incomplete: NVIDIA/numba-cuda-mlir#135's import-time contamination poisons whichever tests reference cuda.cudadrv.driver AFTER the polluting xfail runs, and collection order varies between runs. Two consecutive Windows CI runs failed on different subsets (3 slicing tests one run, 5 interface tests the next). Deselect the full union of #135-listed tests + test_fortran_contiguous (observed to hit the same contamination) on both Linux and Windows. Same version guard (<= 0.4.0) still applies, so the whole block drops automatically when a newer numba-cuda-mlir release ships with the root-cause fix. Linux keeps the extra cuobjdump deselect (Linux-only environment issue). * Revert "cuda_pathfinder: pin nvshmem to <3.7 (was previously excluding only 3.7.0)" This reverts commit 2a42aa7. * Revert "Temporarily add push trigger to ci-nightly.yml for testing" This reverts commit a0ccd19.

Move driver and nvrtc cython and internal layers to new generator

f842321

github-actions Bot added the cuda.bindings Everything related to the cuda.bindings module label Apr 24, 2026

Fix Cython interop tests

992998a

Handle headers differently

6b1d2d3

Fix compilation

be5d972

github-actions Bot added CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module labels Apr 24, 2026

mdboom force-pushed the driver-v2 branch from cbf9660 to be5d972 Compare April 24, 2026 19:26

Make const match

2d7c8dd

Fix compilation again

f664c86

This comment has been minimized.

Sign in to view

leofang assigned mdboom Apr 24, 2026

leofang self-requested a review April 24, 2026 23:59

leofang added this to the cuda.bindings 13.3.0 & 12.9.7 milestone Apr 24, 2026

Attempt to fix Windows build

00ea580

Fix Windows again

9982e06

Fix not found error code

6bf0565

Merge remote-tracking branch 'upstream/main' into driver-v2

103c29f

leofang approved these changes Jun 10, 2026

View reviewed changes

leofang requested changes Jun 11, 2026

View reviewed changes

Comment thread cuda_bindings/cuda/bindings/_internal/driver_linux.pyx Outdated

Comment thread cuda_bindings/cuda/bindings/_internal/driver_linux.pyx Outdated

Merge branch 'main' into driver-v2

0155f57

Fix PTDS and add PTDS testing

7dbeaa0

github-actions Bot added the CI/CD CI/CD infrastructure label Jun 11, 2026

mdboom and others added 3 commits June 11, 2026 13:04

Update .gitignore

3aec179

Co-authored-by: Leo Fang <leof@nvidia.com>

Merge branch 'main' into driver-v2

0a1a49d

Merge branch 'main' into driver-v2

c294d66

mdboom requested a review from leofang June 11, 2026 20:40

mdboom enabled auto-merge (squash) June 11, 2026 20:47

mdboom added 2 commits June 12, 2026 08:24

Add missing 13.3 APIs

13f5106

Merge remote-tracking branch 'origin/driver-v2' into driver-v2

1a87ff3

Correctly update all files

836d5a9

leofang approved these changes Jun 12, 2026

View reviewed changes

mdboom merged commit c128381 into NVIDIA:main Jun 12, 2026
200 of 202 checks passed

This comment has been minimized.

Sign in to view

mdboom mentioned this pull request Jun 12, 2026

Catch up with recent cybind changes #2205

Merged

leofang mentioned this pull request Jun 15, 2026

Backport #1972 to 12.9.x #2206

Merged

leofang reviewed Jul 1, 2026

View reviewed changes

This was referenced Jul 1, 2026

CI: Test CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM=1 #971

Closed

CI: nightly-cuda-core + nightly-numba-cuda-mlir + PTDS fix #2283

Merged

Uh oh!

Conversation

mdboom commented Apr 24, 2026

Uh oh!

copy-pr-bot Bot commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

mdboom commented Apr 24, 2026

Uh oh!

This comment has been minimized.

mdboom commented Apr 25, 2026

Uh oh!

mdboom commented Apr 25, 2026

Uh oh!

mdboom commented Apr 25, 2026

Uh oh!

mdboom commented Apr 29, 2026

Uh oh!

leofang commented Jun 10, 2026

Uh oh!

leofang commented Jun 10, 2026

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mdboom commented Jun 11, 2026

Uh oh!

mdboom commented Jun 11, 2026

Uh oh!

leofang commented Jun 11, 2026

Uh oh!

mdboom commented Jun 12, 2026

Uh oh!

Uh oh!

This comment has been minimized.

github-actions Bot commented Jun 13, 2026

Uh oh!

leofang Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants