In-checkpoint DSparkAttention (shared sliding-window KV) for Qwen3 series — any plans?

[xesdiny]

Hi @sherlockwu — great work on #800 and the V4-Flash DSpark stack.

I have been running standalone DSpark drafter training on Qwen3.5-122B-A10B and hit the structural ceiling that seems hard to escape with the current architecture: the independent draft KV pool consumes ~28% of the KV budget, which causes throughput to fall below vanilla serving above ~50 concurrent requests regardless of accept-length improvements. This problem is compounded on Qwen3+ models that include Mamba/linear-attention layers, where the state cache grows linearly with batch size — adding an independent draft KV pool on top makes the memory pressure even worse at high concurrency.

The V4-Flash-0731 approach in #800 (DSparkAttention with main_x inserted into a bounded sliding-window KV cache) looks like it directly addresses this: the draft KV overhead is fixed rather than proportional to sequence length, so the ~28% KV budget loss disappears.

Question: do you have plans to extend this DSparkAttention mechanism to the Qwen3 GQA family (e.g. Qwen3.5-122B-A10B, Qwen3.8-Flash-Next)? We noticed #812 already notes "Qwen3.5-style preset when a Qwen MoE target is trained" as a follow-up, so wondering if the attention-side sliding-window KV mechanism is also on the roadmap for Qwen targets — or whether you see a blocker we should be aware of.