docs(local-inference): #11391 M6 closure — per-backend kernel decisions, 8/8 verify at shipped pin on M4 Max, kokoro RTF, TurboQuant honest status

[lalalune]

Closes #11391 (Gemma-4 per-platform kernel re-optimization, M6).

What this PR adds

1. Per-backend kernel decision matrix (durable doc) — plugins/plugin-local-inference/native/verify/M6-per-backend-kernel-decisions.md: keep-stock vs re-parameterize for CPU / Metal / Vulkan-desktop / Vulkan-Mali / CUDA / NPU under Gemma-4's dual head dims (512/256) + windowed-SWA + shared-KV, each row grounded in the fork code state at the shipped submodule pin 58c0391eb. Net decision: every llama.cpp backend keeps stock q8_0 KV + FA AUTO (the shipped active-model.ts default); the head_dim=128 QJL/Polar KV kernels are not re-parameterized (dimensionally inapplicable, low ROI per measured geometry) and remain §8-gated for the legacy tiers; NPU is N/A-by-architecture (LiteRT-LM owns its graph).

2. §8 kernel verify matrix re-run per buildable backend at the shipped pin (M4 Max, 2026-07-02) — evidence in .github/issue-evidence/11391-kernel-verify/:

• CPU reference-test: clean.
• Metal: metal-verify 8/8 PASS + metal-verify-shipped (runtime-embedded kernels) + -fused + -multiblock + dispatch-smoke 9/9 graph routes against the freshly built libggml-metal.dylib.
• Vulkan (MoltenVK, device=Apple M4 Max api=1.2.334): vulkan-verify 8/8 PASS.
• CUDA: prior real-hardware 8/8 fixture parity (RTX 5080 sm_120, 2026-06-25, CUDA_VERIFICATION.md); model-backed graph leg hardware-gated → hardware-lab: on-device battery/power/thermal capture + flip mobile-resource-workbench device lanes to gating #11734.
• Vulkan-Mali device 8/8 + NPU delegate verification: hardware-gated → hardware-lab: on-device battery/power/thermal capture + flip mobile-resource-workbench device lanes to gating #11734.

3. Metal FA-engage gap RESOLVED (updates M6-gemma-kv-geometry-and-fa.md): on gemma4-E2B Q8_0 with -fa 1, Metal compiles kernel_flash_attn_ext_f16_dk512_dv512 + dk256_dv256, flash_attn = enabled, 0 padding V cache lines (vs 4 with -fa 0); pp512 2828.6 / tg128 89.7 t/s.

4. Kokoro iSTFTNet RTF re-verified at the shipped pin: one real synth → WAV byte-identical (md5) to the reviewed merged evidence (kokoro-metal-perf/after.wav); 0.90 s user-CPU for 4.7 s audio ≈ compute RTF 0.19 on a load-100+ host (idle-host merged evidence: 723 ms, RTF 0.15; pre-fix 64×). Measured decision recorded: Accelerate/AMX on Apple (fork 114eee08e), threads+NEON portable elsewhere (2bdcef890, PR #11584); a ggml-graph GPU port is now optional, not a blocker.

5. TurboQuant low-precision — honest measured status:

• KV lane (ships as kernels): TBQ4_0 KV on Gemma is quality-parity on CPU (wikitext-2 PPL 101.63 vs 103.54 f16-KV, same chunks) and decodes at 39.4 t/s; prefill ≈0.62× baseline under identical contention. Metal aborts (SET_ROWS on TBQ dst unsupported — Metal whitelists QJL1_256 but no TBQ type; CUDA has both). Gemma runtime decision stays stock q8_0 KV.
• Weight lane: does not exist end-to-end at this pin — no LLAMA_FTYPE_MOSTLY_TBQ*, TBQ3_K/TBQ4_K are CPU-trait-only (zero Metal/Vulkan mul_mat kernels), and the --tensor-type ffn_*=tbq4_k override path fails hard (ggml_validate_row_data: invalid type 50 → quantize failure, measured). Recorded as the honest residual instead of larping a pass. Shipped Gemma weights are K-quants (Q8_0 bundle measured PPL 184.07 vs Q4_K_M requant 204.05, Metal, 8 chunks).

Evidence

.github/issue-evidence/11391-kernel-verify/ — 20 logs + wav + README, every number from output read by hand. Host-load caveat (concurrent agent swarm, load 70–135) annotated on all CPU wall-clock rows.

Docs-and-evidence-only PR: no package source touched; verify/typecheck not applicable (N/A - markdown + committed logs only).

🤖 Generated with Claude Code

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 864eac27-e621-4de3-9f94-e17068893404

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/11391-kernel-verify-closeout

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.