feat: upgrade AIC stack to ROCm 7.14 with vLLM v0.26.0 source build

[sbates130272]

Moves the AIC stack from rocm/dev-ubuntu-24.04:7.2.4-complete to rocm/dev-ubuntu-24.04:7.14.0-full. Replaces the pre-built vLLM wheel with a full source build at v0.26.0. Adds and validates benchmark infrastructure for SPUR cluster (MI355X/gfx950) tiered KV cache.

Stack changes

• Base image: ROCm 7.2.4-complete → 7.14.0-full
• hipFile: removed separate ROCm/rocm-systems build — hipFile is GA in ROCm 7.14 and ships in the base image
• vLLM: pre-built wheel → source build at v0.26.0 with two AMD patches:
  • 01: fix rocm_platform_plugin() — amdsmi returns 0 devices post-torch init; add torch.version.hip fallback for ROCm detection
  • 02: break circular import in rocm.py _get_gcn_arch() — replace logger.warning_once() with stderr print to avoid the vllm.platforms import cycle
• LMCache: v0.5.2 with 9 AMD patches including:
  • 09: serialise NixlStorageAgent.make_prepped_xfer() — the NIXL C++ makeXferReq() is not thread-safe when called concurrently on shared pre-built dlist handles. With max-gpu-workers > 1, multiple asyncio coroutines reach make_prepped_xfer() simultaneously and hit NIXL_ERR_INVALID_PARAM, silently zeroing all L2 reads (agent_rx_bytes_total stays 0). Fix: _xfer_handle_lock serialises handle creation only; in-flight DMA transfers remain parallel.
• Torch: switched to AMD native ROCm 7.14 wheel index (repo.amd.com)
• Runtime env vars added: HSA_ENABLE_IPC_MODE_LEGACY=1, SAFETENSORS_FAST_GPU=1, HIP_FORCE_DEV_KERNARG=1
• Monitoring exporters bumped: node-exporter v1.8.2→v1.12.1, device-metrics-exporter v1.4.2→v1.5.1

SPUR cluster fixes

• NIXL POSIX backend: default AIC_L2_BACKEND=nixl_posix on SPUR — hipFile P2PDMA (AIS_MT) hits NIXL_ERR_BACKEND on XFS LVM nodes because GPUDirect storage is unavailable. POSIX plugin avoids hipFileBufRegister.
• POSIX slot sizing: LMCACHE_NIXL_POSIX_SLOT_SIZE=16777216 (16 MiB) replaces the 256 MiB AIS_MT default, which was 28× too large for POSIX where only actual KV bytes are written per slot (gpt-oss-120b fp8: ~9 MiB per chunk). Fixes 100–400s warmup times per concurrency point.
• lmcache Prometheus port: --prometheus-port 9091 — the default 9090 conflicted with Prometheus itself, silencing all lmcache L1/L2 metrics. prometheus.yml updated to scrape :9091.
• NIXL prefetch cap: --l2-prefetch-max-in-flight 4 — prevents the NIXL descriptor pool from exhausting under concurrent load.
• Post-warmup drain: POST_WARMUP_SLEEP_S=300 (was 5s) — with serialised POSIX writes, the warmup write queue takes hundreds of seconds to flush; 5s left iter 0 measuring write-queue back-pressure not cache hits.
• Monitoring on SPUR: AIC_EXPORTERS=0 — the spur-authz plugin blocks --pid=host, causing the exporter fleet compose-up to fail and take Prometheus down with it. Prometheus-only mode skips the blocked containers.
• sbatch mkdir: _mkdir_shared() helper — NVMe slab dirs on /mnt/m2m_nobackup may be owned by root; falls back to passwordless sudo.
• SPUR sbatch quirks: drop --no-requeue (unsupported), clear --gres=gpu:1 with --gres=, fix AIC_CLIFF_ARMS pass-through.
• Build failure bug: run-build-distribute.sh previously treated Docker buildx failures as cache-lock races and continued, silently shipping the previous tarball. Now exits 1 on any buildx failure.

New Makefile targets

• cliff-spur-l2: SPUR-tuned L2 cliff — per_client prefix, util=0.40, 8 GB DRAM L1, c=32, nvme arm only. Designed to overflow the LMCache DRAM L1 (32 × 630 MiB ≈ 20 GB > 8 GB) and produce ext_hit > 0%.
• cliff-kvd: focused KVD cliff — shared prefix, sparse c ladder (1,8,32,64,128,250), 2 iters.
• $(origin) guards on all per-target variable overrides — fixes silent no-op when Makefile exports empty ?= variables (e.g. VLM_GPU_MEMORY_UTILIZATION) that are set-but-empty in the sbatch shell, causing :-default guards to never fire.

Streaming container logs

docker logs -f --timestamps for lmcache and vllm are now streamed to logs/<job-id>/lmcache/lmcache-stream.log and
logs/<job-id>/vllm/vllm-stream.log throughout each kvd arm, enabling fine-grained timing analysis rather than only capturing logs at teardown.

Validated on SPUR MI355X (gfx950)