Repository navigation
Conversation
build-exporters was submitted as a separate sbatch job which could land on a different SPUR node than the main build. Some nodes (e.g. crsuse2-m2m-185) do not grant the user Docker socket access, causing all three exporter tarballs to silently write 13-byte corrupt files. Move the nvme/rdma/hsa-snoop exporter builds to the END of the build-split and bake remote scripts so they run in the same sbatch allocation on the same node that already proved Docker is accessible. Fall through gracefully (WARN, not error) if Docker is unavailable — the main images are already saved. Also add a pre-flight docker info check at the top of the stand-alone build-exporters script so it fails fast with a clear error message rather than writing corrupt tarballs when the socket is not accessible. Remove the separate build-exporters call from dist-build and dist-build-bake Makefile targets; dist-build-exporters remains for standalone use when needed. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
…paces
spur-authz blocks --pid=host on SPUR compute nodes. Switch to
container-scoped IPC+PID namespace sharing throughout:
- lmcache: ipc:host + pid:host → ipc:shareable (lmcache owns the
shareable namespace; no pid: needed for the owner)
- vllm: defaults ipc/pid:host → service:lmcache (overridable via
VLLM_IPC_MODE / VLLM_PID_MODE env vars)
- node-exporter: pid:host → ${NODEX_PID_MODE:-} so it starts without
spur-authz errors; set NODEX_PID_MODE=host when permitted for
accurate per-process CPU stats
- lmcache-emulator / aic-vllm-emulator: ipc:host → ipc:shareable;
POSIX SHM is visible to joining containers with shareable
- hsa-snoop already used pid:service:lmcache — unchanged
With HSA_ENABLE_IPC_MODE_LEGACY=1 (baked into the image ENV) and
amdgpu-dkms 7.1.3, container-scoped IPC+PID is sufficient for
cross-container HIP KV registration on gfx950 SPUR nodes.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
…phase The monitoring compose starts vllm and hsa-snoop which now default to ipc/pid: service:lmcache, but lmcache is not running during the smoke-test's monitoring-only phase. Set VLLM_IPC_MODE, VLLM_PID_MODE, and AIC_HSA_SNOOP_PID_MODE to "" before start_monitoring in the smoke-test so monitoring containers start without requiring lmcache's namespace. Also add missing hsa_snoop_tar declaration in cmd_test scope. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
… in monitoring-only phase ipc:"" triggers :-default so compose still resolves service:lmcache. Use ipc:none (valid Docker value) for VLLM_IPC_MODE and AIC_HSA_SNOOP_PID_MODE. Change compose pid defaults from :- to - (single-dash) so that an exported empty string VLLM_PID_MODE="" bypasses the service:lmcache default, giving Docker the default isolated PID namespace. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
SPUR's spur-authz plugin blocks several container capabilities needed
by the monitoring stack. This commit works around all known policies
while keeping the stack fully functional on non-SPUR nodes.
Blocked policies and mitigations:
B1 /:/host bind-mount (node-exporter):
- docker-compose.spur.yml overlay replaces /:/host with tmpfs:/host;
CPU/mem/net metrics still flow from /proc+/sys inside the container.
B2 /dev:/dev:rw bind-mount (nvme-exporter):
- SPUR override drops the /dev mount; nvme-cli ioctls unavailable but
the container starts and prometheus scrapes (up=0 expected).
pid:host (hsa-snoop):
- Compose now uses container-scoped pid namespaces (service:lmcache in
normal operation, service:prometheus in the monitoring-only phase via
the SPUR override).
Additional changes:
- All exporters moved to aic-network only (no host port bindings) to
avoid conflicts with system services (e.g. node-exporter on :9100).
Only prometheus retains a host port (9090) for health checks.
- monitoring_healthcheck() rewritten to query prometheus scrape status
(up metric) instead of probing host ports directly; _check_job()
added for per-job prometheus API queries.
- node-exporter added to exporters-safe profile; SPUR override replaces
/:/host with tmpfs so it starts cleanly on SPUR nodes.
- AIC_EXPORTERS defaults to "safe" in smoke-test monitoring phase; safe
mode skips exporters-fabric (nvme) on SPUR via AIC_SPUR_CLUSTER guard
in mon_profile().
- aic_docker_preflight() added to monitoring-lib.sh; called at the start
of every sbatch remote script (build, bake, test, tiny-test, prom-dump,
emulate-test, emulate-mp-test, validate) to detect broken containerd
overlay snapshotters early (missing metadata.db) before wasting time.
- start_monitoring() sweeps leftover monitoring containers before compose
up to prevent port-in-use failures on nodes reused across jobs.
Verified on SPUR (MI355X / gfx950): prometheus, node-exporter, amdgpu-
exporter and hsa-snoop all show up=1; nvme-exporter and rdma-exporter
are WARN as expected (no /dev access, no RDMA hardware on these nodes).
Signed-off-by: Stephen Bates <stephen.bates@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
AITER v0.1.22.post1 JIT-compiles module_aiter_core at vLLM engine startup. The compiler (hipcc) and LLVM toolchain were already present via amdrocm-llvm, but the HIP development headers (hip/hip_runtime.h) were absent from the runtime image, causing: fatal error: 'hip/hip_runtime.h' file not found Diagnosed via a SPUR sbatch diagnostic job that confirmed hipcc at /opt/rocm/core-7.14/bin/hipcc and clang++ at /opt/rocm/core-7.14/lib/llvm/bin/clang++ are both present, but /opt/rocm/include/hip/hip_runtime.h is missing. amdrocm-dev provides the HIP headers required to compile HIP extensions at JIT time, unblocking tiny-test and any other workload that triggers AITER kernel compilation on first use. Signed-off-by: Stephen Bates <stephen.bates@amd.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
AITER JIT pre-requisites (docker/base/Dockerfile):
- Copy full /opt/rocm/include/ tree from the build stage (not just hip/)
so AITER JIT modules beyond module_aiter_core (e.g. module_rmsnorm_quant)
find their required headers (hipcub, rocblas, etc.)
- Symlink /opt/rocm/include -> core-<mm>/include (mirrors lib symlink)
- Symlink clang++/clang as /usr/local/bin/c++/cc so AITER's ninja link
step can find a C++ compiler (c++: not found, exit 127)
FlashAttention build (docker/vllm/Dockerfile):
- Add HIPFLAGS/CXXFLAGS=-Wno-deprecated-attributes to suppress the
composable_kernel deduction-guide deprecation flood that clips the
BuildKit step log at 2 MiB
Tiny-test fixes (.slurm/run-build-distribute.sh):
- KV_TRANSFER_ARG: tcp://aic-lmcache -> tcp://localhost (both vLLM and
lmcache use network_mode:host; Docker DNS does not work on host network)
- AIC_TINY_READY_TIMEOUT: 120 -> 240 iterations (x5s = 20 min) to cover
AITER JIT cold-start compilation on first run
- AITER JIT warmup: after bake completes, load the vllm tarball, run a
GPU container to pre-compile module_aiter_core + module_rmsnorm_quant,
docker commit the compiled .so files back into the image, and re-export
as tarball. Uses SPUR_JOB_GPUS for GPU visibility; always commits even
on partial success (module_aiter_core alone saves ~10s per test run)
Signed-off-by: Stephen Bates <stephen.bates@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
AITER JIT-compiles many HIP kernel modules at vLLM startup (~50-77 min on cold SPUR nodes). Several fixes to handle this reliably: AIC_TINY_READY_TIMEOUT: 240 → 1100 iterations (×5s = ~92 min from loop start) so the health check outlasts AITER JIT on even the slowest nodes. Previous value (20 min) caused near-miss timeouts as vLLM finished startup seconds after the health check expired. Bake AITER warmup: switched from individual aiter.ops.norm import to full vLLM LLM initialization (model='Qwen/Qwen2.5-0.5B-Instruct', enforce_eager=True) so ALL AITER JIT modules needed by vLLM are pre-compiled in the bake. The compiled .so files are committed into the image. Note: compiled modules are node-specific and will recompile on a new node; the warmup ensures at least one node's compilation is baked in, saving time when the test runs on the same node as the build. Warmup now mounts /shared_nfs/huggingface as /hf (HF_HUB_OFFLINE=1) so model loading works without network access during the bake warmup. AIC_ROCM_USE_AITER defaults to 0 for tiny-test; AITER is still imported at startup (vLLM v0.29.0 imports it unconditionally) so this does not reduce JIT time but avoids AITER kernel usage at inference time. Python image caching: loads python:3.12 and python:3.12-slim from shared NFS tarballs before compose up to avoid Docker Hub rate-limit (429). Signed-off-by: Stephen Bates <stephen.bates@amd.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Bates <sbates@raithlin.com>
…ccuracy-test
AITER JIT pre-compilation (patches/aiter + docker/vllm/Dockerfile):
Patch AITER's jit/utils/cpp_extension.py to use PYTORCH_ROCM_ARCH
(gfx950) instead of --offload-arch=native. This allows hipcc to compile
AITER JIT modules at Docker build time — no GPU needed for compilation,
only execution. module_aiter_core compiles in 0.7s at build time; at
runtime JIT warmup is 0.00s and vLLM starts in ~10s vs 77+ min.
Dockerfile: COPY patches/aiter/ and run the patch+precompile script after
aiter wheel install. Compiled .so files baked into image via existing
COPY --from=build_vllm. Also patches FA setup.py for
-Wno-deprecated-attributes to eliminate CK log flood.
BuildKit log limit (docker/buildkitd.toml):
Add buildkitd.toml with log.max-size = -1. Wire into docker buildx
create --config; recreate builder when config is present.
Health check fixes (.slurm/run-build-distribute.sh):
- vLLM image has no curl; replace docker exec curl health checks with
python3 urllib. Both readiness probe and inference request fixed.
- accuracy-test: add aic_docker_preflight to catch broken nodes early.
- accuracy-test: use AIC_VLLM_TARBALL + AIC_LMCACHE_TARBALL (bake
tarballs) instead of monolithic _tarball_path.
dist-build now mirrors dist-build-bake (mk/dist.mk).
accuracy-test fixes (.slurm/run-accuracy.sh):
- Replace all docker exec curl with python3 urllib (curl absent in image).
- _dump_metrics: write to stdout and redirect on host (not to container
filesystem).
- _endpoint_url: use localhost instead of Docker bridge IP (vLLM uses
network_mode:host, has no bridge IP).
- KV_TRANSFER_ARG: tcp://aic-lmcache → tcp://localhost (network_mode:host,
Docker DNS does not resolve container names).
- lmcache metrics port: 8080 → 9091 (8080 is REST API; 9091 is Prometheus).
Verified on SPUR: smoke-test passes, tiny-test passes in ~90s (AITER JIT
pre-compiled + python3 urllib). accuracy-test Phase 1-2 pass (vLLM starts
and scores correctly); Phase 3 NVMe pool growth confirmed (tiering works).
Signed-off-by: Stephen Bates <stephen.bates@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
…uild fix
accuracy-test:
- Set VLM_ENFORCE_EAGER=True for the kvd (LMCache-tiered) arm in
run-accuracy.sh: LMCacheMPConnector registers KV caches after vLLM's
CUDA graph warmup, causing "KV caches are not registered" at inference
time. enforce_eager skips graph capture and sidesteps the ordering issue.
- Wire AIC_TEST_EXCLUDE_NODES into accuracy-test sbatch (was already
supported for build; parity fix).
- Drop NUM_CONCURRENT default from 32 → 4 to avoid overwhelming the
0.5B-Instruct endpoint during the accuracy gate.
- Run lm_eval in a subprocess per call so each gets a clean event loop
(aiohttp TCPConnector holds a reference to the first closed loop and
fails with addr_infos=[] on subsequent calls in the same process).
- Add ${VLM_ENFORCE_EAGER:+--enforce-eager} to docker-compose vllm command.
diagnostics:
- Add docker/lmcache/diag_register_kv_caches.py: instruments
LMCacheMPWorkerAdapter at build time to log transfer_ctx lifecycle
(set / shutdown / None-at-create_recorded_event) with object ids and
stack traces. Helps confirm whether register_kv_caches() hits the
same adapter instance that post_forward() reaches.
build:
- Move inline `python3 -c "import re, glob ..."` FA deprecated-attrs
patch to patches/vllm/02-flash-attn-deprecated-attrs.py. BuildKit
intermittently misparses the multi-line quoted string as a bare
Dockerfile instruction; an external script avoids the parser.
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Bates <sbates@raithlin.com>
#181 (6e324a5) is one commit on top of #184's merge (69d68a4), but it rewrote run-build-distribute.sh from a pre-#184 copy, so every region it left untouched went back to its pre-#184 (9cee154) content. That silently undid #184 here while the other side of each contract (run-accuracy.sh's four split-image vars, spur-dist-build.sh's _err_delegate, the Makefile push-ref help) stayed post-#184. Rerun the merge #181 should have made, keeping our side of every conflict; the three conflicts are resolved in the next commit: git show 9cee154:.slurm/run-build-distribute.sh > base git show 69d68a4:.slurm/run-build-distribute.sh > theirs git merge-file --ours .slurm/run-build-distribute.sh base theirs Base is 9cee154 because a region where our side equals it and 69d68a4 does not is exactly a #184 change that #181 reverted; anything #181 changed itself is left as #181 wrote it. The reapplied hunks come from these #184 branch commits: b6194a1 4a62b89 fc17a3f ERR trap, report once, _aic_err_delegate 3aec0a6 (+2a94904 3e52bec 4ff611a 11d610b 5804915 70ce06a) split images in load/push/all/test, accuracy-test, prometheus-dump, emulate 39db1d2 lmcache wheel in aic-vllm eb0068b aic-base before the emulate image 25dc4f8 per-build cache dirs 6d51b9a smoke-test the lmcache CLI and connector d29a77e serve the built image in emulate tests fcb2548 run the emulator with the HF cache's group #186 already moves accuracy-test to the split tarballs, so that hunk keeps #186's version.
The three regions both PRs rewrote, which the previous commit left on #181's side: - build-emulate goes back to building aic-base in the same job (eb0068b). #181's docker-image://aic-base context names a daemon image the docker-container builder cannot see. - build-split goes back to lmcache, then its wheels export, then vllm (39db1d2, 25dc4f8). #181's separate wheel step had no --cache-from, and the job prunes the builder first, so it recompiled LMCache every run even with a warm cache dir. Its vllm step also still used the shared ${_cache_args}, which the reapplied per-image cache args replace.
aic-image-tag.sh now needs docker buildx (#181), so it can fail where it used to only parse Dockerfiles. Every caller falls back to :latest, so a failure silently renames the images; let its stderr through and keep the fallback. The Makefile and run-build-distribute.sh swallows predate #184; bench.mk's is #181's copy of them.
build-split and bake only removed it after a successful build, so a failed one left a full aic-base layout in the shared $HOME tmp dir. Use the EXIT trap cmd_build's base-first path already has; _sbatch_run runs the body in a subshell, so it cannot clobber the exit-file write.
Phase 1.5 had no --cache-from, and the job prunes the builder first, so every bake recompiled LMCache even with a warm cache dir. Phase 2's lmcache build exports the same build_lmcache_wheel stage with mode=max, so read it, as 25dc4f8's build-split step 3/4 does.
#181 ended four RUN chains with `find ... 2>/dev/null; true`. The `; true` applies to the whole && chain, so a failed vLLM compile, torchvision install or llm-emu import still produced a green layer. It was there because /usr/local/lib/python3 does not exist on Ubuntu 24.04 (pip installs to python3.12), so find always failed. Point find at the dir that exists and confine the tolerance to the cleanup, reporting a failure instead of discarding it.
#181 added a bare import vllm to the build. 5ce9929 removed a vllm import from the build because, on a GPU build host, it selected the ROCm platform and died on the missing render node. Whether a bare import does the same is unverified, but the check only needs the installed version, so read the metadata instead.
39db1d2 made aic-vllm's default stage install aic-lmcache's wheel from a lmcache-wheels build context and fail without one. #181's bake file, written before that, gives vllm only the base context, so `make build` (which bakes `default`) stops at "expected one lmcache wheel". dist-build-bake avoided it by building vllm with plain buildx build. Add a lmcache-wheels target for lmcache's `wheels` stage and pass it to vllm, the bake equivalent of build-split's step 3/4.
${_cache_args} expanded when the job script was rendered, where it is unset, so the run-time assignment was dead and every dist-build-bake recompiled vLLM, AITER and FlashAttention with no cache export (25dc4f8's class). Escape it like the lmcache build's ${_lm_cache_args}.
cmd_bake checked only the lmcache tarball, so a misread job exit could pass with a stale or missing vllm one. Check both, as build-split does.
#181 moved build_nixl, its only consumer, from docker/lmcache into docker/base, but the jobs still pass it only to the lmcache and vllm builds. dist-build-fast therefore built full UCX again. Pass it to build-split's, build-emulate's and bake's base builds.
#181 moved hsa-snoop out of aic-lmcache into its own image, so the check fails on every smoke-test run.
Three more RUN chains end in `... || true`, which covers the whole && chain rather than the step that is allowed to fail: - base build_pytorch: a failed or OOM-killed torch compile produced a green layer, which the mode=max cache then kept serving until build_torchvision failed on the missing wheel. New in #181. - vllm stage: a failed PyPI lmcache install. New in #181. - base system deps: a failed apt-get install. Predates #184. Brace each fallback so it covers only its own command, as aic-vllm's runtime stage already does for the same uninstall.
john00003
requested review from
amd-ivaganev,
gaoikawa,
sbates130272 and
vcave
as code owners
October 23, 2026 23:02
Collaborator
Author
|
/run-ci-accuracy |
john00003
force-pushed
the
users/john00003/reapply-184-fixes
branch
from
October 24, 2026 17:00
309a5f0 to
0499b60
Compare
Collaborator
Author
|
/run-ci-accuracy |
Collaborator
Author
|
/run-ci-accuracy |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
WIP, I will fill this out in more detail later.
Technical Details
Test Plan
Test Result
Submission Checklist