Adds bandwidth.yml playbook for NVIDIA nvbandwidth #834
Conversation
sjpb
changed the title
bandwidth.yml is ran via cudatests.yml
claudia-lola
force-pushed
the
bandwidth-test
branch
from
454756e to
cc61ed3
Compare
| gather_facts: true | ||
| tags: cuda_samples | ||
| tasks: | ||
| - ansible.builtin.import_role: |
There was a problem hiding this comment.
Given we don't even run devicequery, I think we should just remove this task entirely TBH. But leave the role pending thinking more!
ansible/roles/cuda/defaults/main.yml
Outdated
| cuda_persistenced_state: started | ||
| # variables for nvbandwidth (for bandwidth.yml tasks run in cudatests.yml) | ||
| cuda_bandwidth_path: "/var/lib/{{ ansible_user }}/cuda_bandwidth" | ||
| cuda_bandwidth_release_url: "https://github.com/NVIDIA/nvbandwidth/archive/refs/tags/v0.8.tar.gz" |
There was a problem hiding this comment.
I'd rather break the version out here and then use that var in the creates: on the "Download CUDA bandwith test release" task.
|
|
||
| - name: Build CUDA bandwidth test | ||
| ansible.builtin.shell: | ||
| cmd: source /cvmfs/software.eessi.io/versions/2023.06/init/bash && module load Boost/1.82.0-GCC-12.3.0 && . /etc/profile.d/sh.local && cmake .. && make -j {{ ansible_processor_vcpus }} |
There was a problem hiding this comment.
You can do this more readably using one of the many multiline yaml options, e.g.:
| cmd: source /cvmfs/software.eessi.io/versions/2023.06/init/bash && module load Boost/1.82.0-GCC-12.3.0 && . /etc/profile.d/sh.local && cmake .. && make -j {{ ansible_processor_vcpus }} | |
| cmd: >- | |
| source /cvmfs/software.eessi.io/versions/2023.06/init/bash && | |
| module load Boost/1.82.0-GCC-12.3.0 && | |
| . /etc/profile.d/sh.local&& | |
| cmake ..&& | |
| make -j {{ ansible_processor_vcpus }} |
| chdir: "{{ cuda_bandwidth_path }}/nvbandwidth-0.8/build" | ||
| creates: "{{ cuda_bandwidth_path }}/nvbandwidth-0.8/build/nvbandwidth" | ||
|
|
||
| - name: Run CUDA bandwidth test |
There was a problem hiding this comment.
I think this needs changed_when: true to subdue the check error
| @@ -1,3 +1,3 @@ | |||
| --- | |||
| - hosts: cuda | |||
| become: true | |||
There was a problem hiding this comment.
Do we actually need become: true now? See below too re. path.
| group: "{{ ansible_user }}" | ||
| creates: "{{ cuda_bandwidth_path }}/nvbandwidth-0.8" | ||
|
|
||
| - name: Creates CUDA bandwidth test build directory |
There was a problem hiding this comment.
Can you make the name: consistent with the name: on the first task please?
| chdir: "{{ cuda_bandwidth_path }}/nvbandwidth-0.8/build/" | ||
| register: cuda_bandwidth_output | ||
|
|
||
| - name: Save CUDA bandwidth output to bandwidth_results.txt |
There was a problem hiding this comment.
Is there no useful summary we can do here?
| - name: Save CUDA bandwidth output to bandwidth_results.txt | ||
| ansible.builtin.copy: | ||
| content: "{{ cuda_bandwidth_output.stdout }}" | ||
| dest: "{{ appliances_environment_root }}/cudatests/bandwidth_results.txt" |
There was a problem hiding this comment.
When cuda group contains multiple nodes they will all write to the same file.
| register: cuda_bandwidth_output | ||
|
|
||
| - name: Save CUDA bandwidth output to bandwidth_results.txt | ||
| ansible.builtin.copy: |
There was a problem hiding this comment.
Given this is fetching a file, why does this not use ansible.builtin.fetch?
| creates: "{{ cuda_bandwidth_path }}/nvbandwidth-0.8/build/nvbandwidth" | ||
|
|
||
| - name: Run CUDA bandwidth test | ||
| ansible.builtin.shell: | |
There was a problem hiding this comment.
So:
- Rather than using
exportyou can use theenvironmentkeyword - docs - Why do we have to mess with LD_LIBRARY_PATH?
- If we really do, this approach won't work b/c it is e.g. hardcoding the microarch (zen4), which will definitely break (e.g. when using an Intel processor), and versions, which doesn't seem robust.
Is it not sufficent to just activate eessi again? And maybe load some eeesi modules?
2 participants
Modifies
ansible/adhoc/cudatests.ymlto run the NVIDIA nvbandwidth test. This replaces the olderbandwidthTestCUDA Samples utility removed in #687.