Skip to content

[ROCm] update MiniMax-M3 bf16 recipe on docker image and fp8_per_chanel quantization notes #598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hongxiayang wants to merge 2 commits into
base: main
Choose a base branch
from
+33 −6
Open

[ROCm] update MiniMax-M3 bf16 recipe on docker image and fp8_per_chanel quantization notes #598

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
15eee50
[ROCm] update MiniMax-M3 bf16 recipe on docker image and fp8_per_chan…
hongxiayang Jun 29, 2026
6f24579
Merge remote-tracking branch 'upstream/main' into amd-bf16-minimax-m3…
hongxiayang Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 33 additions & 6 deletions models/MiniMaxAI/MiniMax-M3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ meta:
model:
model_id: "MiniMaxAI/MiniMax-M3"
min_vllm_version: "0.24.0"
docker_image:
nightly_required: true
docker_image:
nvidia: "vllm/vllm-openai:minimax-m3"
amd: "vllm/vllm-openai-rocm:minimax-m3"
amd: "vllm/vllm-openai-rocm:nightly"
# Docker-only: MiniMax-M3 support hasn't shipped in a stable wheel, and the
# dedicated image is the supported path — hide the pip tab.
architecture: moe
Expand Down Expand Up @@ -232,19 +233,20 @@ guide: |

### Docker (AMD ROCm)

MiniMax-M3 support has not yet shipped in a stable vLLM release — use the
dedicated Docker image or nightly after the release:
MiniMax-M3 support has not yet shipped in a stable vLLM release (`latest` is
still 0.23.0, with no M3 plugin and no `fp8_per_channel` on ROCm) — use the
nightly image:

```bash
docker pull vllm/vllm-openai-rocm:minimax-m3
docker pull vllm/vllm-openai-rocm:nightly
```

```bash
docker run --rm -it --device /dev/kfd --device /dev/dri --group-add video \
--cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
--shm-size=16g -p 8000:8000 \
--entrypoint /bin/bash \
vllm/vllm-openai-rocm:minimax-m3
vllm/vllm-openai-rocm:nightly
```

## Launching the Server
Expand Down Expand Up @@ -311,6 +313,31 @@ guide: |
--enable-auto-tool-choice
```

Add the vision-encoder flags (`--mm-encoder-tp-mode data
--mm-encoder-attn-backend ROCM_AITER_FA`) for multimodal serving.
Comment on lines +316 to +317

@gemini-code-assist gemini-code-assist Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The inline code block (using backticks) is split across two lines. This can cause markdown parsers to render the newline and leading spaces literally inside the code block, making it difficult to read and copy. It is better to wrap each flag in its own inline code block.

  Add the vision-encoder flags (`--mm-encoder-tp-mode data` and
  `--mm-encoder-attn-backend ROCM_AITER_FA`) for multimodal serving.


#### TP8 + `--quantization fp8_per_channel`

Online **per-channel FP8 (PTPC)** quantization of the BF16 checkpoint halves the
weight footprint (≈100 → **≈50 GiB/GPU**, ~1.75× more KV cache) and lifts
batched throughput with **gsm8k unchanged from BF16** (lossless);
These env vars and the flag are for the **BF16 checkpoint only**.
Comment on lines +321 to +324

@gemini-code-assist gemini-code-assist Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The sentence structure here is slightly awkward due to the semicolon at the end of line 284 followed by a capitalized sentence on line 285. Additionally, there is a trailing space on line 284, and "gsm8k" should be capitalized as "GSM8K" to match the style used elsewhere in the repository.

  Online **per-channel FP8 (PTPC)** quantization of the BF16 checkpoint halves the
  weight footprint (≈100 → **≈50 GiB/GPU**, ~1.75× more KV cache) and lifts
  batched throughput with **GSM8K unchanged from BF16** (lossless).
  These env vars and the flag are for the **BF16 checkpoint only**.


```bash
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=0

vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--block-size 128 \
--attention-backend TRITON_ATTN \
--quantization fp8_per_channel \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```

#### TP8 + Expert Parallel

```bash
Expand Down