RotorQuant / PlanarQuant / IsoQuant extremely slow on Metal with high graph splits (>2)

[lvirbalas]

Hello,

I tested iso3 and polar3 on a build from your llama.cpp fork. I built it with the command provided in README.md for Metal.

The issue is that when loading different models, all of them report:
graph splits >> 2

This usually happens, if any op isn’t fully supported/fused on Metal, or if there’s a missing kernel. Then the compute graph falls back to CPU for parts of it.

Result is very high CPU load, GPU underutilization and prefill speeds <50 t/s, where it should be >300 t/s.

If I run the same models on the original TurboQuant fork with turbo3, they all nicely show:
graph splits = 2
(And don't have CPU utilization)

Is this issue expected and simply not implemented yet? Or is it actually a bug to dig into further?

Hardware: MacBook Pro M3 Max

[joostwestra]

Confirmed too high graph splits on M1 macbook and vibe coded a fix (so a full PR might not be what you, but I added the patch here as well)

AI summery:

Problem
Deferred KV key conversion was only effectively reachable for direct llama_kv_cache usage. On the tested Metal path, llama_context held a wrapper memory type, so the concrete cast failed and deferred K remained F16 while V was iso4.

That left Flash Attention seeing a mixed F16 K + iso4 V state and produced repeated CPU attention islands, with sched_reserve: graph splits = 14.

Fix
expanded the deferred-conversion gate to include Metal
replaced the concrete cast in llama_context with a virtual llama_memory_i::convert_deferred_keys()
forwarded that hook through wrapper memory implementations:
hybrid
iSWA
hybrid+iSWA
Result
On the tested macOS/Metal setup with iso4 KV cache:

deferred K conversion now executes correctly
scheduler rebuilds around quantized K tensors
graph splits drop from 14 to 2

Files changed
src/llama-context.cpp
src/llama-memory.h
src/llama-kv-cache.h
src/llama-kv-cache-iswa.h
src/llama-kv-cache-iswa.cpp
src/llama-memory-hybrid.h
src/llama-memory-hybrid.cpp
src/llama-memory-hybrid-iswa.h
src/llama-memory-hybrid-iswa.cpp

metal-deferred-k-conversion.patch

[joostwestra]

speed difference: 172.15 -> 310.05. about +80.1%:

Here’s the true iso4/iso4 + FA before/after
Same command, two builds:

bash
./build/bin/llama-bench
-m /Users/joostwestra/models/Qwen3.5-0.8B-Q4_K_S.gguf
-ngl 99 -fa 1 -ctk iso4 -ctv iso4 -pg 512,128 -r 5

Before
Build: fc3d1b656
pp512: 1135.18 ± 67.33 t/s
tg128: 56.62 ± 1.86 t/s
pp512+tg128: 172.15 ± 21.67 t/s
After
Build: d6d96b52f
pp512: 1127.08 ± 88.25 t/s
tg128: 75.29 ± 1.27 t/s
pp512+tg128: 310.05 ± 6.17 t/s
Delta
Prefill: essentially flat / slightly down
1135.18 -> 1127.08
about -0.7%
Decode: strong improvement
56.62 -> 75.29
about +33.0%
Combined: very strong improvement
172.15 -> 310.05
about +80.1%