Recover from the nvidia-cufile "undefined symbol: shm_open" load failure - #2537
Open
LeSingh1 wants to merge 1 commit into
Open
Recover from the nvidia-cufile "undefined symbol: shm_open" load failure#2537LeSingh1 wants to merge 1 commit into
LeSingh1 wants to merge 1 commit into
Conversation
nvidia-cufile wheels reference shm_open() but carry no DT_NEEDED entry for
librt. On glibc < 2.34, where shm_open still lives in librt rather than in
libc, dlopen() therefore fails and load_nvidia_dynamic_lib("cufile") raises:
RuntimeError: Failed to dlopen .../nvidia/cu13/lib/libcufile.so.0:
.../libcufile.so.0: undefined symbol: shm_open
Loading librt.so.1 with RTLD_GLOBAL beforehand makes the reference resolve,
which is the workaround kvikio carries today (rapidsai/kvikio#1000).
Apply it as a strictly reactive retry rather than an unconditional preload:
nothing happens unless the primary load has already failed with exactly this
signature (libname "cufile" plus "shm_open" in the dlopen error). That keeps
the normal path byte-for-byte unchanged, keeps librt out of the global symbol
scope for everyone else, and makes the workaround disappear by itself once the
wheel is fixed. If the retry does not help, the original error is reported,
since it is the more informative one.
Adds tests/test_load_dl_linux.py covering the retry, the two ways it can give
up, and that neither an unrelated cufile failure nor the same symptom on a
different library triggers it.
Refs NVIDIA#2313
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the failure reported in #2313.
nvidia-cufilewheels referenceshm_open()but ship noDT_NEEDEDentry for librt.On glibc < 2.34 — where
shm_openstill lives inlibrtrather than in libc —dlopen()therefore fails andload_nvidia_dynamic_lib("cufile")dies with:Loading
librt.so.1withRTLD_GLOBALfirst makes the reference resolve. That is theworkaround kvikio carries today (rapidsai/kvikio#1000), and the one @gforsyth wrote up
in the issue.
Shape of the fix
The obvious implementation is an unconditional preload in
_work_around_known_bugs(),next to the existing
nvrtcentry. I did not do that, because it would put librt intothe global symbol scope of every process that loads cufile, forever, including the
overwhelming majority where nothing is broken.
Instead the workaround is strictly reactive: it runs only after the primary
dlopen()has already failed, and only when the failure carries exactly this signature(libname
cufileandshm_openin the dlopen error).That buys three things:
dlopenon success;future regression (the concern already written into the
nvrtcworkaround's comment);the more informative one.
Scope notes
load_with_abs_path()is covered. That is the path taken for the wheel installin the report (found via site-packages). I deliberately left
load_with_system_search()alone rather than widen the blast radius beyond the reported symptom — happy to extend
it if you'd rather have both.
standing dependency of cufile, so it does not belong in
dependencies.Tests
New
cuda_pathfinder/tests/test_load_dl_linux.py(the Linux-side counterpart totest_load_dl_windows.py). No real cufile install is needed —_load_libandctypes.CDLLare stubbed, so the tests reproduce the exact failure signaturedeterministically. Coverage:
main);main);main);shm_opensymptom on a different library,do not trigger the workaround — these two pass on
mainas well, on purpose; theyare there to pin down that the trigger stays narrow.
Verified: 3 of the 5 fail against
upstream/mainand all 5 pass with the change; therest of
cuda_pathfinder/testshas the same pass/fail set asmain.ruff check,ruff format --checkclean, andmypyreports no new errors (the one pre-existingLoadedDLarg-typeerror in this file is untouched).One question for you: #2313 is assigned to @juenglin — if that work is already in
flight, say the word and I'll close this.
Refs #2313