Skip to content

fix(kimi-k3): a DSpark draft's MLA cache cannot diverge from the target's - #1015

Open
torchspec-bot wants to merge 1 commit into
mainfrom
fix/k3-dspark-mla-cache-dtype
Open

fix(kimi-k3): a DSpark draft's MLA cache cannot diverge from the target's#1015
torchspec-bot wants to merge 1 commit into
mainfrom
fix/k3-dspark-mla-cache-dtype

Conversation

@torchspec-bot

@torchspec-bot torchspec-bot commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Since #974 unified draft KV into the target's cache pool, an MLA draft owns no pool of its own: create_cache_pool only accepts a backing view for ordinary MHA pools, so _create_draft_components hands a DSpark draft a LayerMappedKVPool onto the target's buffer — and that pool binds every latent_kv field with a single store dtype.

resolve_mla_kv_cache_dtype still pins a K3 DSpark draft to BF16 whenever the target runs FP8. Its docstring's premise — "the draft owns a separate pool" — predates #974 and is no longer true. The K3 recipe then plans the draft's continuation planes at the draft's itemsize while the pool binds them at the target's, so a K3 DSpark launch on the tokenspeed_mla backend — which hard-requires --kv-cache-dtype fp8_e4m3 (tokenspeed_mla.py:191) — dies at startup:

ValueError: field 'layer.93.latent_kv': dtype itemsize does not match plan

This makes the draft follow the target, and rewrites the docstring to record why BF16 isn't reachable from here.

Trade-off worth flagging. BF16 is the faithful choice: the public DSpark checkpoint ships no FP8 KV scales, and the reference vLLM launch uses a BF16 cache. Following the target means FP8 context injection, and the engine duly logs Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. In our runs acceptance did not visibly suffer, but this is a real numeric difference, not a no-op. Restoring BF16 needs heterogeneous MLA backing views — a larger change than this regression fix should carry.

One question for reviewers: resolve_mla_kv_cache_dtype is now a pure passthrough to resolve_dtype. I kept the function because the docstring is the whole value — it's the tombstone explaining why the BF16 path is unreachable. Happy to inline it and delete the function (plus the now-vacuous test_other_mla_drafts_keep_the_requested_cache_dtype) if you'd rather.

Test Plan

…et's

Since #974 unified draft KV into the target's cache pool, an MLA draft owns no
pool of its own. create_cache_pool only accepts a backing view for ordinary MHA
pools, so _create_draft_components hands a DSpark draft a LayerMappedKVPool onto
the target's buffer -- and that pool binds every latent_kv field with a single
store dtype.

resolve_mla_kv_cache_dtype still pinned the draft to BF16 whenever the target
ran K3's required FP8 LCM cache; its docstring's premise -- that the draft owns
a separate pool -- predates #974. The K3 recipe then planned the draft's
continuation planes at the draft's itemsize while the pool bound them at the
target's, so a K3 DSpark launch on the tokenspeed_mla backend, which
hard-requires --kv-cache-dtype fp8_e4m3, dies at startup with

    ValueError: field 'layer.93.latent_kv': dtype itemsize does not match plan

The BF16 preference is the faithful one -- the public DSpark checkpoint has no
FP8 KV scales and the reference vLLM launch uses a BF16 cache -- but it is not
reachable from here. Follow the target and say so; restoring BF16 needs
heterogeneous MLA backing views.

Signed-off-by: torchspec-bot <262938024+torchspec-bot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant