Skip to content

Recover from the nvidia-cufile "undefined symbol: shm_open" load failure - #2537

Open
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:cufile-librt-workaround
Open

Recover from the nvidia-cufile "undefined symbol: shm_open" load failure#2537
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:cufile-librt-workaround

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Closes the failure reported in #2313.

nvidia-cufile wheels reference shm_open() but ship no DT_NEEDED entry for librt.
On glibc < 2.34 — where shm_open still lives in librt rather than in libc —
dlopen() therefore fails and load_nvidia_dynamic_lib("cufile") dies with:

RuntimeError: Failed to dlopen .../nvidia/cu13/lib/libcufile.so.0:
.../libcufile.so.0: undefined symbol: shm_open

Loading librt.so.1 with RTLD_GLOBAL first makes the reference resolve. That is the
workaround kvikio carries today (rapidsai/kvikio#1000), and the one @gforsyth wrote up
in the issue.

Shape of the fix

The obvious implementation is an unconditional preload in _work_around_known_bugs(),
next to the existing nvrtc entry. I did not do that, because it would put librt into
the global symbol scope of every process that loads cufile, forever, including the
overwhelming majority where nothing is broken.

Instead the workaround is strictly reactive: it runs only after the primary
dlopen() has already failed, and only when the failure carries exactly this signature
(libname cufile and shm_open in the dlopen error).

That buys three things:

  • the normal path is byte-for-byte unchanged — no extra dlopen on success;
  • the workaround disappears by itself once the wheel is fixed, so it cannot mask a
    future regression (the concern already written into the nvrtc workaround's comment);
  • if the retry does not help, the original error is what the user sees, since it is
    the more informative one.

Scope notes

  • Only load_with_abs_path() is covered. That is the path taken for the wheel install
    in the report (found via site-packages). I deliberately left load_with_system_search()
    alone rather than widen the blast radius beyond the reported symptom — happy to extend
    it if you'd rather have both.
  • No descriptor/catalog change: this is a defect in one specific wheel build, not a
    standing dependency of cufile, so it does not belong in dependencies.

Tests

New cuda_pathfinder/tests/test_load_dl_linux.py (the Linux-side counterpart to
test_load_dl_windows.py). No real cufile install is needed — _load_lib and
ctypes.CDLL are stubbed, so the tests reproduce the exact failure signature
deterministically. Coverage:

  • the retry succeeds after librt is loaded (fails on main);
  • the original error is reported when the retry still fails (fails on main);
  • the retry is skipped when librt itself cannot be loaded (fails on main);
  • an unrelated cufile failure, and the same shm_open symptom on a different library,
    do not trigger the workaround — these two pass on main as well, on purpose; they
    are there to pin down that the trigger stays narrow.

Verified: 3 of the 5 fail against upstream/main and all 5 pass with the change; the
rest of cuda_pathfinder/tests has the same pass/fail set as main. ruff check,
ruff format --check clean, and mypy reports no new errors (the one pre-existing
LoadedDL arg-type error in this file is untouched).

One question for you: #2313 is assigned to @juenglin — if that work is already in
flight, say the word and I'll close this.

Refs #2313

nvidia-cufile wheels reference shm_open() but carry no DT_NEEDED entry for
librt. On glibc < 2.34, where shm_open still lives in librt rather than in
libc, dlopen() therefore fails and load_nvidia_dynamic_lib("cufile") raises:

    RuntimeError: Failed to dlopen .../nvidia/cu13/lib/libcufile.so.0:
    .../libcufile.so.0: undefined symbol: shm_open

Loading librt.so.1 with RTLD_GLOBAL beforehand makes the reference resolve,
which is the workaround kvikio carries today (rapidsai/kvikio#1000).

Apply it as a strictly reactive retry rather than an unconditional preload:
nothing happens unless the primary load has already failed with exactly this
signature (libname "cufile" plus "shm_open" in the dlopen error). That keeps
the normal path byte-for-byte unchanged, keeps librt out of the global symbol
scope for everyone else, and makes the workaround disappear by itself once the
wheel is fixed. If the retry does not help, the original error is reported,
since it is the more informative one.

Adds tests/test_load_dl_linux.py covering the retry, the two ways it can give
up, and that neither an unrelated cufile failure nor the same symptom on a
different library triggers it.

Refs NVIDIA#2313
@copy-pr-bot

copy-pr-bot Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.pathfinder Everything related to the cuda.pathfinder module label Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.pathfinder Everything related to the cuda.pathfinder module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant