ROCm AMD Infinity Context

Caution

This release is an early-access software technology preview. Running production workloads is not recommended.

ROCm(tm) AMD Infinity Context (AIC) is a disaggregated KV-cache inference stack for large language models on AMD Instinct GPUs. It combines an LLM serving framework with a KV Cache block manager to offload KV tensors across a tiered memory hierarchy. GPU VRAM, CPU DRAM, local NVMe and NFS-over-RDMA.

It focuses on enabling a low-latency, shared level of KV Cache that can be accessed by O(1000) GPUs.

ROCm AIC enables long-context serving at scale without recomputation. This repository provides a Docker image build and test harness, benchmarking harness, and Slurm/Spur automation used to validate and demonstrate the platform.

Stack Overview

Component	Source	Ref
Base OS	rocm/dev-ubuntu-24.04:7.14.1-full	Ubuntu 24.04, ROCm 7.14, Python 3.12
PyTorch	ROCm/pytorch (source build)	2.13 (ec5f3b1)
vLLM	github.com/vllm-project/vllm (source build)	v0.29.0 + 3 AMD patches
AITER	ROCm/aiter (source build)	v0.1.22.post1 (vLLM ROCm-validated)
FlashAttention	Dao-AILab/flash-attention (source build)	0e60e394 (vLLM ROCm-validated)
LMCache	LMCache/LMCache (upstream)	v0.5.6 + 16 AMD patches
NIXL	ai-dynamo/nixl (upstream)	v1.4.1 + nixl-rocm-ais-mt.patch
hsa-snoop	sbates130272/hsa-snoop (source build)	v1.2.0
hipFile	ROCm 7.14 base image	GA in ROCm 7.14 — no separate source build

Pip Nightly Wheels

See docs/PIP_WHEELS.md for installation instructions and compatibility notes. Wheels are rebuilt nightly from main and published to the nightly release.

Prerequisites

• ROCm-capable host. Pass ROCM_ARCH as a ;-separated list (e.g. gfx90a;gfx942;gfx950) to build a multi-arch image, or a single arch (e.g. ROCM_ARCH=gfx942) for a faster build. The vLLM source build compiles GPU kernels for exactly the archs specified.
• Docker with BuildKit and the docker compose (v2) plugin (Docker 23+). On a node that lacks it, make ensure-compose installs the plugin user-locally (~/.docker/cli-plugins); the Slurm cliff / smoke / tiny-test jobs self-install it automatically.
• Host mounts: local NVMe (NVME_DATA) and NFS-over-RDMA (NFS_DATA) pre-mounted
• HuggingFace token with access to the target model
• Python 3.10+ for host-side benchmarks

Quick Start

See docs/QUICK_START.md for the full step-by-step guide (build, start, benchmark, plot). For Slurm / SPUR cluster usage see docs/SLURM_SPUR.md.

Metrics & Observability

See docs/METRICS_TELEMETRY.md for the full Prometheus scrape config, exporter details, NIXL telemetry, hsa-snoop, and NFS caveats.

Key Environment Variables

See docs/ENVIRONMENT.md for the full reference.

License

MIT