Added NCCL flags from DPO #1165
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
finbarrtimbers
requested review from
saurabh111233212 and
tyler-romero
and removed request for
saurabh111233212
| # Add COLL here to log all collective operations. Extreamly verbose, dont use for production. | ||
| beaker.BeakerEnvVar(name="NCCL_DEBUG_SUBSYS", value="INIT,NET"), | ||
| beaker.BeakerEnvVar( | ||
| name="LD_LIBRARY_PATH", value=f"/var/lib/tcpxo/lib64:{os.getenv(LD_LIBRARY_PATH, '')}" |
Contributor
There was a problem hiding this comment.
LGTM but I'm also seeing these from DPO:
--env NCCL_PROTO=Simple,LL128 \
--env NCCL_TUNER_CONFIG_PATH=/var/lib/tcpxo/lib64/a3plus_tuner_config_ll128.textproto \
--env NCCL_SHIMNET_GUEST_CONFIG_CHECKER_CONFIG_FILE=/var/lib/tcpxo/lib64/a3plus_guest_config_ll128.textproto \
I'm not sure they're necessary, especially the NCCL_PROTO one, but just flagging for awareness.
We're also running source /var/lib/tcpxo/lib64/nccl-env-profile.sh before launching the actual accelerate command.
Contributor
There was a problem hiding this comment.
Ah I see you mentioned that the other flags are already defaults
Collaborator
Author
There was a problem hiding this comment.
Yes, these are all set (except for nccl-env-profile.sh). I'm not sure how to source nccl-env-profile.sh programmatically. Do you have any ideas?
tyler-romero
approved these changes
Nov 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These worked for DPO!
The other flags are already defaults. With this, we should be able to remove the
--envflags for all the DPO scripts, as all the defaults will be correct.Note
Adds
LD_LIBRARY_PATHandNCCL_LIB_DIRenv vars to the GCP cluster NCCL configuration inmason.py.get_env_varsGCP branch, add:LD_LIBRARY_PATHto include/var/lib/tcpxo/lib64plus existing value.NCCL_LIB_DIRset to/var/lib/tcpxo/lib64.Written by Cursor Bugbot for commit a07f8d9. This will update automatically on new commits. Configure here.