fix(layers): pass a 0-token batch through SiluAndMul without a launch - #1077
Open
slin1237 wants to merge 1 commit into
Open
fix(layers): pass a 0-token batch through SiluAndMul without a launch#1077slin1237 wants to merge 1 commit into
slin1237 wants to merge 1 commit into
Conversation
A DP rank's idle forward (ForwardMode.IDLE) runs the model over a
0-token batch so the rank stays in the group's collectives. Any model
whose MLP uses the fused SiluAndMul then launches flashinfer's
silu_and_mul over an empty grid, and the launch itself fails:
RuntimeError: Check failed: (err == cudaSuccess) is false:
Failed to launch kernel: invalid argument
killing the scheduler mid-lockstep. Reproduced deterministically with
Qwen3-4B at dp=2 under an external ZMQ frontend on H100 (two
consecutive engine starts died at the first idle forward); Llama passes
only because its path never enters this kernel with zero rows.
An empty input fully determines the output, so return the empty output
(and, for fp8_out, the zero-row TMA-aligned scale the fused path would
collapse to) before any kernel launch. The non-empty path is unchanged.
Tests: empty-batch passthrough for both the plain and fp8_out forms
(shape/dtype/device, scale shape mirroring the fused path's zero-row
collapse), plus a 1-token eager-reference check that the guard did not
disturb the computed path.
Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4fd4ad0d27
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # copies of the Software, and to permit persons to whom the Software is | ||
| # furnished to do so, subject to the following conditions: | ||
| # | ||
| # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
There was a problem hiding this comment.
This new file's license block skips the MIT condition clause between the grant and warranty disclaimer, so it does not satisfy the repository requirement to use the full MIT header for copyright notices; add the missing condition paragraph before the disclaimer.
AGENTS.md reference: AGENTS.md:L15-L16
Useful? React with 👍 / 👎.
slin1237
added a commit
to smg-project/smg
that referenced
this pull request
Aug 13, 2026
### Problem The pinned TokenSpeed cannot run Qwen3 under DP: a rank's idle forward pushes a 0-token batch through the model, and Qwen3's MLP launches flashinfer's silu_and_mul over an empty grid — the launch fails with CUDA invalid argument and kills the scheduler mid-lockstep. Four out of four Qwen3 group starts died this way across three CI runs, and every subsequent test against the dead group burned its client timeout, so the whole e2e-2gpu-chat-zmq-dp (tokenspeed) lane stays red on one engine bug. ### Solution Deselect the affected model on exactly that lane — TokenSpeed runtime AND a grouped ZMQ worker — at collection, next to the existing ZMQ family filter. Every other lane keeps Qwen3: single-engine TokenSpeed runs it fine, and vLLM DP runs it fine. The engine fix is lightseekorg/tokenspeed#1077 (SiluAndMul passes a 0-row batch through without a kernel launch); the skip carries that pointer and dies with the pin bump that adopts it. ## Changes - `e2e_test/fixtures/hooks.py`: `_TOKENSPEED_DP_BROKEN_MODELS` + `_filter_tokenspeed_dp_items`, applied only when `get_runtime() == "tokenspeed"` and `get_zmq_engine_count() > 1`. - `e2e_test/infra/__init__.py`: export `get_zmq_engine_count`. ## Test Plan Collection verified against the tier-1 chat suite in all three neighboring configurations: - tokenspeed + zmq + engine count 2: `TestToolChoiceQwen` deselected (0 items), 268 collected otherwise unchanged. - tokenspeed + zmq + engine count unset: Qwen3 kept (1 class) — the single-engine lane is unaffected. - vllm + zmq + engine count 2: Qwen3 kept — the filter is runtime-scoped, not lane-wide. Live check is this PR's e2e-2gpu-chat-zmq-dp (tokenspeed) lane, which should now be green on the Llama suite that already passes. Signed-off-by: Simo Lin <25425177+slin1237@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A DP rank's idle forward (
ForwardMode.IDLE) runs the model over a 0-token batch so the rank stays in the group's collectives. Any model whose MLP uses the fusedSiluAndMulthen launches flashinfer'ssilu_and_mulover an empty grid, and the launch itself fails:killing the scheduler mid-lockstep; the sibling rank then dies on the broken collective (gloo "connection closed by peer"). Reproduced deterministically with Qwen3-4B at
dp=2under an external ZMQ frontend on H100 — four out of four group starts died at the first idle forward. Llama passes only because its path never enters this kernel with zero rows.Solution
An empty input fully determines the output, so
SiluAndMul.forwardreturns the empty output (and, forfp8_out, the zero-row TMA-aligned scale the fused path would collapse to) before any kernel launch. The non-empty path is unchanged.Tests
test_silu_and_mul_passes_through_empty_batch/..._fp8_...: empty-batch passthrough for both forms — shape, dtype, device, and the scale shape mirroring the fused path's zero-row collapse. Both pass on GB300.test_silu_and_mul_nonempty_still_computes: a 1-token batch against the eager reference, guarding that the guard did not disturb the computed path.