feat(docker): ROCm ASAN nan-repro image (hipBLASLt 1.4.0)#287
Open
amd-vivekag wants to merge 1 commit into
Open
feat(docker): ROCm ASAN nan-repro image (hipBLASLt 1.4.0)#287amd-vivekag wants to merge 1 commit into
amd-vivekag wants to merge 1 commit into
Conversation
Adds Dockerfile.rocm70_2-nan-asan, which layers ROCm's ASAN-instrumented libraries onto the customer-aligned nan-repro hipBLASLt 1.4.0 base for running the residual-NaN repro under AddressSanitizer. The base is Ubuntu 24.04 (noble) but ROCm only publishes *-asan packages for jammy (22.04), so the Dockerfile extracts the jammy *-asan leaf packages' .so files into /opt/rocm/lib/asan (dpkg-deb -x, no install) to avoid clashing with / clobbering the custom noble ROCm build. Requires --network=host at build time. Wires the variant into .env.example, README (variant row, Example 5, file structure) and setup-env.sh (menu option 6). Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new ROCm 7.0.2 Docker variant intended to run the residual-NaN hipBLASLt 1.4.0 repro under AddressSanitizer, and wires it into the existing docker workflow/documentation.
Changes:
- Introduces
Dockerfile.rocm70_2-nan-asanthat downloads/extracts jammy*-asanROCm leaf packages onto the noble-based nan-repro image and configures runtime env (LD_PRELOAD,ASAN_OPTIONS, etc.). - Updates the interactive docker env setup script to include the new variant.
- Updates docker docs and templates (
README.md,.env.example) to document/configure the new Dockerfile.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| docker/Dockerfile.rocm70_2-nan-asan | New Dockerfile that layers ROCm ASAN libraries and sets ASAN-related runtime configuration. |
| docker/setup-env.sh | Adds a menu option/variant selection for the new ASAN image. |
| docker/README.md | Documents the new Dockerfile and provides an example configuration/build snippet. |
| docker/.env.example | Lists the new Dockerfile variant in the template comments. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # first in initial library list" workaround. HSA_XNACK requests GPU-side | ||
| # (device) ASAN; on an xnack- target it degrades gracefully to host-side | ||
| # coverage only. | ||
| ENV LD_LIBRARY_PATH=/opt/rocm/lib/asan:/opt/rocm/lib |
Comment on lines
+54
to
+55
| apt-get download -o Acquire::AllowInsecureRepositories=true --allow-unauthenticated ${ASAN_PACKAGES}; \ | ||
| for d in *.deb; do echo "unpacking $d"; dpkg-deb -x "$d" /; done; \ |
Comment on lines
+134
to
+136
| Then drive the residual-NaN matrix under ASAN via the sidecar in the internal | ||
| repo (`recipes/recom_repro_residual_nan_asan_side_car.json`); see the | ||
| "run under ROCm ASAN" section of `docs/Residual-NaN-Repro-runbook.md`. |
Comment on lines
+51
to
+54
| echo "deb [arch=amd64 trusted=yes] ${ASAN_APT_URL} ${ASAN_APT_DIST} main" \ | ||
| > /etc/apt/sources.list.d/rocm-asan.list; \ | ||
| apt-get update -o Acquire::AllowInsecureRepositories=true; \ | ||
| apt-get download -o Acquire::AllowInsecureRepositories=true --allow-unauthenticated ${ASAN_PACKAGES}; \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docker/Dockerfile.rocm70_2-nan-asan: layers ROCm's ASAN-instrumented libraries onto the customer-aligned nan-repro hipBLASLt 1.4.0 base, for running the residual-NaN repro under AddressSanitizer.*-asanpackages for jammy (22.04). So the Dockerfile downloads only the jammy*-asanleaf packages anddpkg-deb -xextracts their.sos into/opt/rocm/lib/asan— no dpkg DB changes, no dependency resolution — leaving the custom noble ROCm 7.0.2 build-56 stack (incl. the customlibhipblaslt.so.1.4.70002) untouched. Build number70002-56matches the base exactly..env.example,README.md(variant row, Example 5, file structure) andsetup-env.sh(menu option 6).Build
Host networking is required (default docker bridge has no outbound net on the build box):
Test plan
/opt/rocm/lib/asan, ASAN env vars set (HSA_XNACK=1,LD_PRELOAD→clang_rt.asan,ASAN_OPTIONS), base ROCm + custom hipBLASLt 1.4.0 intact.import torchunder the preloaded ASAN runtime does not hit "ASan runtime does not come first".AddressSanitizer: out of memoryinlibamdhip64.so.7) because the stock jammy asan libs over a non-instrumented PyTorch aren't device-ASAN-capable on this host. Needs an ASAN-instrumented PyTorch/HIP stack (tracked separately). Documented in the internal runbook.Companion internal PR adds the sidecar + runbook section.
Made with Cursor