[Issue]: CE collectives fail if they are run on default cuda stream

[ngoyal2707]

How is this issue impacting you?

Application crash

Share Your Debug Logs

cudaMemcpyBatchAsync fails when default stream is passed with invalid attribute error.

pytorch latest now defaults to using user stream for when async_op=False. So when we use ce_collectives with pytorch in async_op=False mode, default stream gets passed to cudaMemcpyBatchAsync and NCCL fails with RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)

Maybe it cant be solved in NCCL but if so, might be better to handle this gracefully by raising some error about the stream can't be default stream.

Steps to Reproduce the Issue

No response

NCCL Version

2.29.2+cuda13.1

Your platform details

No response

Error Message & Behavior

No response

[teojgo]

@ngoyal2707 do you have a reproducer for this problem?

[kwen2501]

Thanks for trying the feature.
It is a known limitation that NCCL CE collectives does not work with default CUDA stream as of 2.28.
Please refer to the following documentation in PyTorch on how to work around this limitation:

[ngoyal2707]

Thanks @kwen2501 for sharing the PT docs link, I have been also forcing the non default stream and it works.

The issue was mainly that maybe we can detect the cuda_stream being default sooner and fail with better error message in NCCL.

[nvcastet]

A WAR is to set the CE threshold (default to 8MiB) to a large value, aka NCCL_SYM_CE_THRESHOLD=1000000000000