cuda.core: validate ctypes host callback signatures against CUhostFn (#2525)

* cuda.core: validate ctypes host callback signatures against CUhostFn

Reject incompatible ctypes prototypes before CUDA sees them, document
the required ABI, and note the stronger checking in the 1.2.0 release notes.

* cuda.core: make ctypes flag lookups stubgen/mypy-friendly

Use getattr for private ctypes calling-convention constants so the
regenerated _host_callback.pyi type-checks cleanly.

* cuda.core: check host callback prototypes via public ctypes attributes

The previous check inspected ctypes' private _flags_ bits to identify the
calling convention. That is wrong on Windows: CPython defines
FUNCFLAG_STDCALL as 0, so a bitwise test can never match WINFUNCTYPE, and
every win-64 test job rejected a valid callback. The 0x2 fallback used when
_ctypes.FUNCFLAG_STDCALL is absent is FUNCFLAG_HRESULT, not stdcall.

Drop the calling-convention check rather than repair the bit arithmetic.
ctypes only honors stdcall when building a callback on 32-bit x86 Windows,
which cuda.core does not support, and FUNCFLAG_PYTHONAPI is never consulted
on the callback path, so CFUNCTYPE, WINFUNCTYPE, and PYFUNCTYPE all yield the
same FFI_DEFAULT_ABI thunk. That leaves the declared result and argument
types, which are reachable through the public restype/argtypes attributes.

Reading those public attributes also lets a function pointer taken from a
shared library be accepted once its restype and argtypes are declared, which
the class-level lookup could never see.

Author: Andy-Jost

cuda_core/cuda/core/graph/_graph_builder.pyi

@@ -409,10 +409,12 @@ class GraphBuilder:
         - **Python callable**: Pass any callable. The GIL is acquired
           automatically. The callable must take no arguments; use closures
           or ``functools.partial`` to bind state.
-        - **ctypes function pointer**: Pass a ``ctypes.CFUNCTYPE`` instance.
-          The function receives a single ``void*`` argument (the
-          ``user_data``). The caller must keep the ctypes wrapper alive
-          for the lifetime of the graph.
+        - **ctypes function pointer**: The function receives a single
+          ``void*`` argument (the ``user_data``), and the caller must keep
+          the ctypes wrapper alive for the lifetime of the graph. Its
+          declared prototype must match the driver's ``CUhostFn``
+          (``void (*)(void*)``): ``ctypes.CFUNCTYPE(None, ctypes.c_void_p)``,
+          or ``ctypes.WINFUNCTYPE(None, ctypes.c_void_p)`` on Windows.
 
         .. warning::
 
@@ -432,6 +434,14 @@ class GraphBuilder:
             Only for ctypes function pointers. If ``int``, passed as a raw
             pointer (caller manages lifetime). If bytes-like, the data is
             copied and its lifetime is tied to the graph.
+
+        Raises
+        ------
+        TypeError
+            If ``fn`` is a ctypes function pointer whose declared prototype
+            does not match ``CUhostFn``.
+        ValueError
+            If ``user_data`` is given for a Python callable.
         """
 
 class Graph:

cuda_core/cuda/core/graph/_graph_builder.pyx

@@ -860,10 +860,12 @@ cdef class GraphBuilder:
         - **Python callable**: Pass any callable. The GIL is acquired
           automatically. The callable must take no arguments; use closures
           or ``functools.partial`` to bind state.
-        - **ctypes function pointer**: Pass a ``ctypes.CFUNCTYPE`` instance.
-          The function receives a single ``void*`` argument (the
-          ``user_data``). The caller must keep the ctypes wrapper alive
-          for the lifetime of the graph.
+        - **ctypes function pointer**: The function receives a single
+          ``void*`` argument (the ``user_data``), and the caller must keep
+          the ctypes wrapper alive for the lifetime of the graph. Its
+          declared prototype must match the driver's ``CUhostFn``
+          (``void (*)(void*)``): ``ctypes.CFUNCTYPE(None, ctypes.c_void_p)``,
+          or ``ctypes.WINFUNCTYPE(None, ctypes.c_void_p)`` on Windows.
 
         .. warning::
 
@@ -883,6 +885,14 @@ cdef class GraphBuilder:
             Only for ctypes function pointers. If ``int``, passed as a raw
             pointer (caller manages lifetime). If bytes-like, the data is
             copied and its lifetime is tied to the graph.
+
+        Raises
+        ------
+        TypeError
+            If ``fn`` is a ctypes function pointer whose declared prototype
+            does not match ``CUhostFn``.
+        ValueError
+            If ``user_data`` is given for a Python callable.
         """
         GB_callback(self, fn, user_data, False)
 

cuda_core/cuda/core/graph/_graph_node.pyi

@@ -333,10 +333,12 @@ class GraphNode:
         - **Python callable**: Pass any callable. The GIL is acquired
           automatically. The callable must take no arguments; use closures
           or ``functools.partial`` to bind state.
-        - **ctypes function pointer**: Pass a ``ctypes.CFUNCTYPE`` instance.
-          The function receives a single ``void*`` argument (the
-          ``user_data``). The caller must keep the ctypes wrapper alive
-          for the lifetime of the graph.
+        - **ctypes function pointer**: The function receives a single
+          ``void*`` argument (the ``user_data``), and the caller must keep
+          the ctypes wrapper alive for the lifetime of the graph. Its
+          declared prototype must match the driver's ``CUhostFn``
+          (``void (*)(void*)``): ``ctypes.CFUNCTYPE(None, ctypes.c_void_p)``,
+          or ``ctypes.WINFUNCTYPE(None, ctypes.c_void_p)`` on Windows.
 
         .. warning::
 
@@ -361,6 +363,14 @@ class GraphNode:
         -------
         HostCallbackNode
             A new HostCallbackNode representing the callback.
+
+        Raises
+        ------
+        TypeError
+            If ``fn`` is a ctypes function pointer whose declared prototype
+            does not match ``CUhostFn``.
+        ValueError
+            If ``user_data`` is given for a Python callable.
         """
 
     def if_then(self, condition: GraphCondition) -> IfNode:

cuda_core/cuda/core/graph/_graph_node.pyx

@@ -490,10 +490,12 @@ cdef class GraphNode:
         - **Python callable**: Pass any callable. The GIL is acquired
           automatically. The callable must take no arguments; use closures
           or ``functools.partial`` to bind state.
-        - **ctypes function pointer**: Pass a ``ctypes.CFUNCTYPE`` instance.
-          The function receives a single ``void*`` argument (the
-          ``user_data``). The caller must keep the ctypes wrapper alive
-          for the lifetime of the graph.
+        - **ctypes function pointer**: The function receives a single
+          ``void*`` argument (the ``user_data``), and the caller must keep
+          the ctypes wrapper alive for the lifetime of the graph. Its
+          declared prototype must match the driver's ``CUhostFn``
+          (``void (*)(void*)``): ``ctypes.CFUNCTYPE(None, ctypes.c_void_p)``,
+          or ``ctypes.WINFUNCTYPE(None, ctypes.c_void_p)`` on Windows.
 
         .. warning::
 
@@ -518,6 +520,14 @@ cdef class GraphNode:
         -------
         HostCallbackNode
             A new HostCallbackNode representing the callback.
+
+        Raises
+        ------
+        TypeError
+            If ``fn`` is a ctypes function pointer whose declared prototype
+            does not match ``CUhostFn``.
+        ValueError
+            If ``user_data`` is given for a Python callable.
         """
         return GN_callback(self, fn, user_data)
 

cuda_core/cuda/core/graph/_host_callback.pyi

@@ -1,3 +1,19 @@
 # This file was generated by stubgen-pyx v0.2.6 from cuda_core/cuda/core/graph/_host_callback.pyx
 
-from __future__ import annotations
+from __future__ import annotations
+
+import sys
+
+_CUHOSTFN_HINT = 'ctypes.CFUNCTYPE(None, ctypes.c_void_p)' if sys.platform != 'win32' else 'ctypes.CFUNCTYPE(None, ctypes.c_void_p) or ctypes.WINFUNCTYPE(None, ctypes.c_void_p)'
+
+def _cuhostfn_type_error(detail):
+    """Build the rejection message for a non-conforming ctypes callback."""
+
+def _validate_ctypes_host_callback(fn):
+    """Reject ctypes callbacks whose declared prototype is not CUhostFn.
+
+    ``restype`` and ``argtypes`` are the prototype the caller declared, and are
+    what CUDA calls through. A function pointer taken from a shared library
+    keeps ctypes' defaults -- a ``c_int`` result and unspecified arguments --
+    until the caller declares otherwise, so it must be declared to be accepted.
+    """

cuda_core/cuda/core/graph/_host_callback.pyx

@@ -14,9 +14,47 @@ from cuda.core._resource_handles cimport (
     make_opaque_py,
 )
 
+import sys
 import ctypes as ct
 
 
+# CUhostFn is `void (CUDA_CB *)(void*)`. CUDA_CB is __stdcall on Windows and
+# empty elsewhere, but ctypes only honors that distinction when it builds a
+# callback on 32-bit x86 Windows, which cuda.core does not support: on win-64
+# and ARM64 both CFUNCTYPE and WINFUNCTYPE produce a FFI_DEFAULT_ABI thunk. The
+# declared result and argument types are all that remain worth checking.
+_CUHOSTFN_HINT = (
+    "ctypes.CFUNCTYPE(None, ctypes.c_void_p)"
+    if sys.platform != "win32"
+    else "ctypes.CFUNCTYPE(None, ctypes.c_void_p) or "
+    "ctypes.WINFUNCTYPE(None, ctypes.c_void_p)"
+)
+
+
+def _cuhostfn_type_error(detail):
+    """Build the rejection message for a non-conforming ctypes callback."""
+    return TypeError(
+        f"host callback {detail}; CUDA requires a callback matching CUhostFn "
+        f"(void (*)(void*)), declared as {_CUHOSTFN_HINT}. "
+        "Alternatively, pass a Python callable."
+    )
+
+
+def _validate_ctypes_host_callback(fn):
+    """Reject ctypes callbacks whose declared prototype is not CUhostFn.
+
+    ``restype`` and ``argtypes`` are the prototype the caller declared, and are
+    what CUDA calls through. A function pointer taken from a shared library
+    keeps ctypes' defaults -- a ``c_int`` result and unspecified arguments --
+    until the caller declares otherwise, so it must be declared to be accepted.
+    """
+    restype = fn.restype
+    argtypes = fn.argtypes
+    if restype is not None or argtypes is None or tuple(argtypes) != (ct.c_void_p,):
+        raise _cuhostfn_type_error(
+            f"has prototype restype={restype!r}, argtypes={argtypes!r}")
+
+
 cdef void _py_host_trampoline(void* data) noexcept with gil:
     (<object>data)()
 
@@ -36,8 +74,12 @@ cdef void _resolve_host_callback(
     ``cuGraphAddHostNode`` or ``cuLaunchHostFunc``. ``*out_fn_owner`` owns the
     callback object; ``*out_data_owner`` owns a copied ``user_data`` buffer and
     is left null otherwise. The caller attaches both owners to the graph node.
+
+    ctypes callbacks are validated against the ``CUhostFn`` ABI before their
+    address is passed to CUDA.
     """
     if isinstance(fn, ct._CFuncPtr):
+        _validate_ctypes_host_callback(fn)
         out_fn[0] = <cydriver.CUhostFn><uintptr_t>ct.cast(fn, ct.c_void_p).value
         if user_data is None:
             out_user_data[0] = NULL

cuda_core/cuda/core/graph/_subclasses.pyi

@@ -320,6 +320,11 @@ class HostCallbackNode(GraphNode):
     def update(self, fn, *, user_data=None) -> None:
         """Replace the callback and user-data binding for this node.
 
+        ``fn`` accepts the same forms as :meth:`~graph.GraphNode.callback`: a
+        Python callable, or a ctypes function pointer whose declared prototype
+        matches ``CUhostFn`` (``void (*)(void*)``). A mismatched ctypes
+        prototype raises ``TypeError``.
+
         .. warning::
 
             Callbacks must not call CUDA API functions. Doing so may
@@ -508,6 +513,10 @@ class ExecutableHostCallbackNode(ExecutableGraphNode):
     def update(self, fn, *, user_data=None) -> None:
         """Replace the callback and user-data binding for future launches.
 
+        ``fn`` may be a Python callable, or a ctypes function pointer whose
+        declared prototype matches ``CUhostFn`` (``void (*)(void*)``); a
+        mismatched prototype raises ``TypeError``.
+
         .. warning::
 
             Callbacks must not call CUDA API functions. Doing so may deadlock

cuda_core/cuda/core/graph/_subclasses.pyx

@@ -1172,6 +1172,11 @@ cdef class HostCallbackNode(GraphNode):
     def update(self, fn, *, user_data=None) -> None:
         """Replace the callback and user-data binding for this node.
 
+        ``fn`` accepts the same forms as :meth:`~graph.GraphNode.callback`: a
+        Python callable, or a ctypes function pointer whose declared prototype
+        matches ``CUhostFn`` (``void (*)(void*)``). A mismatched ctypes
+        prototype raises ``TypeError``.
+
         .. warning::
 
             Callbacks must not call CUDA API functions. Doing so may
@@ -1603,6 +1608,10 @@ cdef class ExecutableHostCallbackNode(ExecutableGraphNode):
     def update(self, fn, *, user_data=None) -> None:
         """Replace the callback and user-data binding for future launches.
 
+        ``fn`` may be a Python callable, or a ctypes function pointer whose
+        declared prototype matches ``CUhostFn`` (``void (*)(void*)``); a
+        mismatched prototype raises ``TypeError``.
+
         .. warning::
 
             Callbacks must not call CUDA API functions. Doing so may deadlock

cuda_core/docs/source/release/1.2.0-notes.rst

@@ -59,6 +59,20 @@ Fixes and enhancements
   (`#2517 <https://github.com/NVIDIA/cuda-python/pull/2517>`__,
   closes `#2516 <https://github.com/NVIDIA/cuda-python/issues/2516>`__)
 
+- ``cuda.core`` now checks ctypes host callbacks against the driver's
+  ``CUhostFn`` signature (``void (*)(void*)``) before passing the function
+  pointer to CUDA. :meth:`graph.GraphNode.callback`,
+  :meth:`graph.GraphBuilder.callback`, and the host-callback ``update()``
+  methods raise ``TypeError`` for a mismatched prototype, rather than leaving
+  the driver to call through an incompatible signature, which is undefined
+  behavior. Declarations that previously reached the driver, such as
+  ``ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p)``, are now rejected at the
+  call site. A function pointer obtained from a shared library keeps ctypes'
+  default ``c_int`` result type until it is declared, so set its ``restype``
+  and ``argtypes`` (or cast it to the prototype above) before passing it. On
+  Windows, both ``ctypes.CFUNCTYPE`` and ``ctypes.WINFUNCTYPE`` are accepted.
+  (`#2439 <https://github.com/NVIDIA/cuda-python/issues/2439>`__)
+
 Deprecation Notices
 -------------------
 

cuda_core/tests/graph/test_graph_builder.py

@@ -306,6 +306,21 @@ def read_byte(data):
     assert result[0] == 0xAB
 
 
+@pytest.mark.agent_authored(model="cursor-grok-4.5")
+def test_graph_capture_callback_ctypes_rejects_incompatible_signature(init_cuda):
+    """Stream-capture host callbacks use the same ctypes ABI check."""
+    import ctypes
+
+    bad_type = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p)
+    launch_stream = Device().create_stream()
+    gb = launch_stream.create_graph_builder().begin_building()
+    try:
+        with pytest.raises(TypeError, match="CUhostFn"):
+            gb.callback(bad_type(0))
+    finally:
+        gb.end_building()
+
+
 @pytest.mark.agent_authored(model="claude-opus-4.8")
 def test_graph_capture_callback_python_survives_del(init_cuda):
     """Captured callback is retained by its graph-node user object after del."""