Skip to content

Fix L2 persistence Modal CI test#50

Merged
msaroufim merged 1 commit into
masterfrom
codex/fix-l2-persistence-modal-ci
Jun 9, 2026
Merged

Fix L2 persistence Modal CI test#50
msaroufim merged 1 commit into
masterfrom
codex/fix-l2-persistence-modal-ci

Conversation

@msaroufim

Copy link
Copy Markdown
Member

Summary

Move the L2 persistence test's CUDA device queries, libcudart loading, and large weight allocation behind lazy process-local initialization. This avoids initializing CUDA at module import before the isolated benchmark subprocess/thread runs.

Root cause

The Modal L4 CI job failed in test/l2_persistence.py when torch.mv(_W, x) raised CUBLAS_STATUS_INVALID_VALUE. The test created _W and queried CUDA globals at import time, then the benchmark harness spawned an isolated process and ran setup inside the native benchmark thread. Deferring that CUDA state creation keeps the test setup inside the process that uses it.

Validation

  • python3 -m py_compile test/l2_persistence.py ci/modal_gpu_test.py python/pygpubench/__init__.py
  • git diff --check
  • Reran the same Modal L4 test command using the failed workflow wheel artifact; l2_persistence.py completed with hacked/honest median ratio = 1.001 and OK: L2 persistence confers no benchmark advantage.

@msaroufim
msaroufim marked this pull request as ready for review June 9, 2026 04:09
@msaroufim
msaroufim merged commit 7b2ef7f into master Jun 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant