Support for NIXL push mode

[iyastreb]

Purpose

Recently NIXL push mode was added to vLLM, before that only NIXL pull existed (RDMA READ).
vllm-project/vllm#35264
vllm-project/vllm#36923

vllm-router used to work with NIXL pull mode (sequential), and in this mode new PUSH mode is slower than PULL mode, which is unexpected. It happens because of the extra message (see RFC)

Scenario 1: D registers before P finishes (overlap)
D sends REGISTER_BLOCKS_MSG to P's scheduler via ZMQ side channel
P stores the registration data
When P finishes the request (request_finished), it finds the stored registration
P enqueues to the push dispatcher thread, which sends a PUSH_TRIGGER_MSG via ZMQ TCP to all TP workers
Scenario 2: P finishes before D registers (push message D->P)
P finishes the request and stores block IDs in _finished_request_blocks
When D's REGISTER_BLOCKS_MSG arrives at P's listener thread, it enqueues to the push dispatcher
The dispatcher fuzzy-matches the registration to the finished request and sends the push trigger

The key insight is that vllm-router (sequential) always goes with scenario 2 for push mode, meaning extra message from D to P, which is why PUSH performance drops compared to PULL.

The fix is to make router parallel for push mode: process_concurrent_two_stage fires prefill and decode with a single call — both HTTP requests leave the router at the same time, and neither waits for the other.

Impl details:

1. Router learns prefill details from the first response and cache them, so no bootstrap or extra config needed, router works with --kv-connector nixl
2. On vLLM side push connector needs to be started with new config option transfer_mode:push:
   I think we can commit tiny PR to vLLM to automatically add this option for Push connector

diff --git a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/push_scheduler.py b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/push_scheduler.py
index 23c6e6cd3..2ee803433 100644
--- a/vllm/distributed/kv_transfer/kv_connector/v1/nixl/push_scheduler.py
+++ b/vllm/distributed/kv_transfer/kv_connector/v1/nixl/push_scheduler.py
@@ -286,6 +286,7 @@ class NixlPushConnectorScheduler(NixlBaseConnectorScheduler):
             remote_port=self.side_channel_port,
             tp_size=self.vllm_config.parallel_config.tensor_parallel_size,
             remote_num_tokens=remote_num_tokens,
+            transfer_mode="push",
         )
 
     def build_connector_meta(

Test Plan

Tested with benchmark

OUT=vllm.jsonl && rm -f "$OUT" && \
for num_prompts in 1 2 4 8 16 32 64 128 256 512 1024; do
  echo "========== num-prompts: $num_prompts =========="
  for run in 1 2 3 4 5; do
    vllm bench serve --host 127.0.0.1 --port 8200 \
      --model $MODEL --num-prompts $(( num_prompts * 2 )) \
      --max-concurrency $num_prompts \
      --random-input-len 1024 --random-output-len 64 \
      --random-range-ratio 0 --seed 42 \
      --num-warmups 8 --dataset-name random \
      --save-result --append-result --result-filename "$OUT" \
      --disable-tqdm 2>/dev/null \
      | awk -v r=$run '
          /Median TTFT/ {ttft=$NF}
          /P99 TTFT/    {ttft99=$NF}
          /Median TPOT/ {tpot=$NF}
          /P99 TPOT/    {printf "  run %s:  TTFT med=%s (p99 %s) ms   TPOT med=%s (p99 %s) ms\n", r, ttft, ttft99, tpot, $NF}'
  done
  echo ""
done

Test Result

# p4d4, best mean TTFT of 4 runs, vllm-router
num_prompts  Mooncake  NIXL-PULL  NIXL-PUSH  NIXL-PUSH-PARALLEL
1            75        74         73         71
2            112       108        133        96
4            136       137        147        120
8            170       159        184        123
16           184       183        198        159
32           231       191        226        182
64           235       204        237        210
128          287       248        270        216
256          366       373        392        372
512          2398      2095       2171       2087

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2352c9fca

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".