[AgentX] vLLM DeepSeek-V4 B200 aggregate MTP#2259
Conversation
…tion #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r every agentic arm
Add an MTP speculative-decoding twin (spec-decoding: mtp,
num_speculative_tokens=3) for every arm of dsv4-fp4-b200-vllm-agentic
(TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each
mirroring its non-MTP conc-list, routed via the launcher's
spec-decoding=mtp suffix to dsv4_fp4_b200_vllm_mtp.sh.
New dsv4_fp4_b200_vllm_mtp.sh forks dsv4_fp4_b200_vllm.sh with only the MTP
deltas: --speculative-config {"method":"mtp","num_speculative_tokens":3} and
--max-cudagraph-capture-size scaled to MAX_NUM_SEQS*(1+N) tokens so
FULL_DECODE_ONLY still covers the largest decode batches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Document the dsv4-fp4-b200-vllm-agentic MTP twins (num_speculative_tokens=3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29565769390 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --speculative-config "{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}" | ||
| --no-disable-hybrid-kv-cache-manager | ||
| --disable-uvicorn-access-log | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --max-num-seqs "$MAX_NUM_SEQS" |
There was a problem hiding this comment.
🔴 The new agentic script benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh runs real (non-synthetic) MTP decoding instead of the fairness-mandated synthetic acceptance pinned to the golden AL curve. Its --speculative-config only sets method/num_speculative_tokens, missing rejection_sample_method=synthetic and synthetic_acceptance_length (2.49 thinking_on / 2.97 thinking_off from golden_al_distribution/dsv4_mtp.yaml), unlike the precedent kimik2.5_fp4_b300_mtp.sh script.
Extended reasoning...
The bug: docs/PR_REVIEW_CHECKLIST.md contains a mandatory, agentic-scoped CODEOWNER sign-off item (line 23):
For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in
golden_al_distribution/for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
golden_al_distribution/README.md's "Fairness Guidelines for AgentX" section reinforces this and shows the exact vLLM mechanism: inject the golden value via rejection_sample_method: "synthetic" + synthetic_acceptance_length: <value> in --speculative-config. The committed curve at golden_al_distribution/dsv4_mtp.yaml gives deepseek-v4-pro, num_speculative_tokens=3 → AL 2.49 (thinking_on) / 2.97 (thinking_off).
Where it breaks: In the new file benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh (line ~260), the --speculative-config passed to vllm serve is:
--speculative-config "{\"method\": \"mtp\", \"num_speculative_tokens\": $NUM_SPEC_TOKENS}"This has no rejection_sample_method or synthetic_acceptance_length field, so vLLM performs real (non-synthetic) MTP verification against the actual draft model's predictions rather than sampling acceptance at a fixed, pinned rate.
Why nothing else catches it: The only other agentic MTP/EAGLE script in the repo, benchmarks/single_node/agentic/kimik2.5_fp4_b300_mtp.sh, correctly follows the fairness contract — it sets rejection_sample_method:"synthetic" and synthetic_acceptance_length (2.88 / 3.24, selected by DCP_SIZE branch) directly in its --speculative-config string. The new dsv4 script has no equivalent branch or env-var override anywhere in the file that injects these fields; VLLM_CMD is built once, inline, with the literal string above, so there is no other code path that could compensate.
Impact: Without pinning acceptance to the golden curve, the measured acceptance length (and therefore decode throughput/speedup from MTP) is free to drift with whatever the real draft model happens to achieve on the agentic replay traffic. This defeats the stated purpose of the golden-AL mechanism — making agentic MTP speedup numbers comparable across submissions/vendors — and is exactly the failure mode the checklist item and Fairness Guidelines were written to prevent. It is also a mandatory, explicit CODEOWNER sign-off checklist item, not a style preference.
Step-by-step proof:
- A reviewer runs the new pure-TP+MTP tier added in
configs/nvidia-master.yaml(spec-decoding: mtp,conc-list: [1,2,4,6,8]). dsv4_fp4_b200_vllm_mtp.shlaunchesvllm servewith--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'.- vLLM's MTP draft head proposes tokens and the target model verifies them for real — acceptance depends on true model/draft agreement on the actual agentic trace content, which is not controlled.
- Compare to
kimik2.5_fp4_b300_mtp.sh: its config additionally carries"rejection_sample_method":"synthetic","synthetic_acceptance_length":2.88, forcing every decode step to accept exactly the pinned AL regardless of true draft/target agreement. - Because dsv4's config lacks those two fields, its published throughput/TPOT numbers reflect whatever real AL the traces produce that day/run, not the committed
2.49/2.97golden value — so DeepSeek-V4 MTP results are not apples-to-apples comparable to other vendors' agentic MTP submissions the way the fairness guideline requires.
Fix: Add rejection_sample_method: "synthetic" and synthetic_acceptance_length: <2.49 or 2.97> (selected based on thinking mode, mirroring how kimik2.5_fp4_b300_mtp.sh branches on DCP_SIZE) to the --speculative-config JSON at line ~260.
Note: fixed_seq_len/dsv4_fp4_b200_vllm_mtp.sh also uses real decoding, but the checklist item is explicitly scoped to agentic workloads, so that script is out of scope for this finding.
| search-space: | ||
| # Pure TP at low concurrency. | ||
| - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] } | ||
| # Pure TP GPU-resident + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8] } | ||
| - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] } | ||
| # Pure TP SimpleCPU + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, spec-decoding: mtp, conc-list: [8, 12, 16] } | ||
| # DEP | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } } | ||
| # DEP SimpleCPU + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, spec-decoding: mtp, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } } | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } } | ||
| # DEP Mooncake + MTP (num_speculative_tokens=3) | ||
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, spec-decoding: mtp, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } } | ||
|
|
||
| dsv4-fp4-b200-trt: |
There was a problem hiding this comment.
🔴 The PR adds 6 new MTP search-space rows (and the new dsv4_fp4_b200_vllm_mtp.sh script) to dsv4-fp4-b200-vllm-agentic in configs/nvidia-master.yaml, but never appends the required entry to perf-changelog.yaml. Since .github/workflows/run-sweep.yml gates both its push and pull_request triggers on paths: perf-changelog.yaml, the benchmark sweep will never trigger for this PR, leaving the new MTP arms unbenchmarked and unvalidated by CI. Please append a config-keys: [dsv4-fp4-b200-vllm-agentic] entry to perf-changelog.yaml before merging.
Extended reasoning...
What's missing
This PR introduces 6 new MTP-variant agentic search-space rows under dsv4-fp4-b200-vllm-agentic in configs/nvidia-master.yaml (MTP counterparts of the pure-TP, SimpleCPU-offload, and DEP/Mooncake tiers), plus the new benchmarks/single_node/agentic/dsv4_fp4_b200_vllm_mtp.sh script that actually runs them. However, the PR touches only three files — the new script, configs/nvidia-master.yaml, and utils/matrix_logic/validation.py — and never appends an entry to perf-changelog.yaml.
Why this matters (not just a doc nit)
AGENTS.md's "Adding a benchmark configuration" section is explicit: "Add entries to configs/nvidia-master.yaml ..., append to perf-changelog.yaml, then validate...". That alone would just be a process nit, except perf-changelog.yaml is not merely documentation — it is the mechanism that drives CI. .github/workflows/run-sweep.yml gates both its push (to main) and pull_request triggers on:
paths:
- "perf-changelog.yaml"
I verified this directly in the workflow file (lines ~22-36). Since this PR never modifies perf-changelog.yaml, the check-changelog/validate-perf-changelog job and the downstream benchmark-sweep job never run for this PR at all — not "run and pass trivially," but never invoked in the first place.
Proof / concrete walkthrough
- I checked the tail of
perf-changelog.yaml: the most recent entries reference PR [AgentX] Update vLLM DeepSeek-V4 B300 aggregate / 更新 vLLM DeepSeek-V4 B300 聚合配置 #2241 (dsv4-fp4-b300-vllm-agentic), Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试 #2228, and [NV] Add GLM-5 NVFP4 GB200 disaggregated Dynamo TensorRT-LLM MTP benchmarks / [NV] 新增 GLM-5 NVFP4 GB200 分离式 Dynamo TensorRT-LLM MTP 基准测试 #1800 — there is no entry for this PR or its new MTP config keys. run-sweep.yml'son.pull_request.pathsandon.push.pathsboth list onlyperf-changelog.yaml.- GitHub Actions path-filtered triggers only fire the workflow when a commit in the PR touches a file matching that path. This PR's diff (
dsv4_fp4_b200_vllm_mtp.sh,configs/nvidia-master.yaml,utils/matrix_logic/validation.py) matches none of those paths. - Therefore
run-sweep.ymldoes not trigger on this PR's pushes/synchronize events, socheck_changelog/process_changelog.pynever runs, never discovers the 6 new MTP rows (it discovers rows to benchmark by diffingperf-changelog.yamlbetween base and head), and the sweep infrastructure that would exercisedsv4_fp4_b200_vllm_mtp.shnever executes. - Net effect: the new MTP arms — the entire purpose of this PR — go unbenchmarked and unvalidated by CI, silently. Nothing fails loudly; the workflow simply never starts.
Precedent
Every comparable recent PR in this repo followed the append-to-changelog step: #2224 (dsv4-fp4-b200-vllm-agentic), #2241 (dsv4-fp4-b300-vllm-agentic), #2228 (kimik2.5-fp4-b300-vllm-agentic-mtp), and #1800 (glm5-fp4-gb200-dynamo-trt-mtp) all appended a perf-changelog.yaml entry alongside their config changes. This PR breaks that established, repository-mandated pattern.
Fix
Append a new entry to the end of perf-changelog.yaml, e.g.:
- config-keys:
- dsv4-fp4-b200-vllm-agentic
description:
- "Add MTP (num_speculative_tokens=3) variants of the pure-TP, SimpleCPU-offload, and DEP/Mooncake agentic tiers for B200 AgentX."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2259This is not a stylistic ask — it is the mechanism by which the sweep workflow actually triggers, so omitting it defeats the PR's own verification.
Restructure so the sweep runs only the new MTP work, not the existing aggregate: - Revert dsv4-fp4-b200-vllm-agentic to its main (#2224) search space -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b200-vllm-agentic-mtp: MTP twins (num_speculative_tokens=3) of the aggregate arms (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list. - Point the perf-changelog entry at only the new key, so only it sweeps. (No separate TP8 key: B200's TP8 arms already exist in the #2224 aggregate.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use MTP synthetic rejection sampling with acceptance length 2.49, the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) from golden_al_distribution/dsv4_mtp.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-mtp # Conflicts: # perf-changelog.yaml
Lower gpu-memory-utilization for the B200 vLLM MTP recipe to 0.9 for extra headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29566610531 |
1 participant
No description provided.