Skip to content

fix(parallel_read): pin reader threads to the destination device - #46

Merged
jetjodh merged 1 commit into
mainfrom
fix/reader-thread-device
Oct 9, 2026
Merged

jetjodh merged 1 commit into
mainfrom
fix/reader-thread-device

Conversation

@jetjodh

@jetjodh jetjodh commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every process that loads a flashpack onto cuda:N (N > 0) through the parallel reader leaves a stray ~600 MiB CUDA context on GPU 0. In an 8-rank job, that's seven extra contexts, about 4.3 GiB of GPU 0's memory, so rank 0 runs out of memory before its peers.

Reader threads start on cuda:0. Each one copies inside
`torch.cuda.stream(stream)`, and leaving that context restores the thread's
previous stream, which is cuda:0's, and switches the thread to device 0. On
CUDA 12+ that switch initializes device 0's primary context. So every
process that loads onto cuda:N (N > 0) leaves a ~600 MiB context on GPU 0.
In an 8-rank job that is 7 extra contexts, ~4.3 GiB of GPU 0's memory, and
rank 0 runs out of memory before its peers.

Set the reader thread's device to the destination device first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jetjodh
jetjodh merged commit d1db2a8 into main Oct 9, 2026
6 checks passed
@jetjodh
jetjodh deleted the fix/reader-thread-device branch October 9, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant