Gemma 4 cutover + multi-backend libelizainference (LiteRT/AICore/CoreML/MLX) + per-platform kernel optimization

[lalalune]

Gemma 4 cutover + multi-backend libelizainference — living plan

Status doc for the Gemma 4 cutover campaign. Supersedes the research issue
#8794 (Gemma is real + shipped; the open questions there are answered below).
Continuously updated as milestones land.

Goal

Cut eliza-1 over from the Qwen3.5/3.6 backbone to Gemma 4 entirely
(remove Qwen), and make every platform run the fastest in-process backend
behind one FFI pipe (libelizainference):

Backend	Where	§11 status
llama.cpp	everywhere (CPU / CUDA / Vulkan-Mali-Adreno / Metal)	owned pipe (today)
LiteRT-LM (compiled-in)	Android NPU (Tensor / Qualcomm QNN / MediaTek NeuroPilot), opt. desktop/iOS GPU	owned pipe ✅
CoreML / MLX (compiled-in)	Apple (mac first, iOS later)	owned pipe ✅
AICore / Apple Foundation	opportunistic fast-path	external (not owned)

§11 ("single on-device runtime") is reinterpreted from "llama.cpp-only" to
"one managed library, one pipe, no sidecar/subprocess/TCP." LiteRT-LM and
MLX are embeddable in-process C++ libraries → linked into libelizainference,
exposing the same FFI streaming symbols FfiStreamingRunner already drives.
AICore is an out-of-process Android system service (Binder IPC) → it stays
an opportunistic adapter like apple-foundation.ts, never the owned backend.

Gemma 4 — validated facts (2026)

Shipped 2026-04-02 (MTP drafters 04-16, 12B Unified 06-03).

• Sizes: E2B (~2.3B eff / 5.1B w/ embeddings), E4B (~4.5B / 8B), 12B Unified
  (11.95B dense, encoder-free, 256K ctx, 262,144 vocab), 31B dense, 26B-A4B
  (MoE, 4B active). E2B ≈ 1.3 GB Q4, 2–3 GB RAM → mobile tier.
• Architecture: dense; alternating local sliding-window (512/1024) + global
  full-attention; dual RoPE; Per-Layer Embeddings (PLE); shared KV cache
  (last N layers reuse earlier KV). No SSM/Gated-DeltaNet. Dual head dims
  (global_head_dim + SWA head_dim).
• MTP: official separate drafter models (speculative decode ~3×, no quality
  loss). Our catalog's drafterFile? already supports separate-drafter MTP.
• Modalities: image + audio (USM conformer) + video.
• NPU fit: dense graphs are NPU-delegate-friendly; LiteRT-LM ships
  pre-converted Gemma 4 .litertlm bundles. Qwen3.5-hybrid is hostile to NPU
  delegates — the strongest technical reason to cut over.
• Our llama.cpp fork already speaks Gemma 4: LLM_ARCH_GEMMA4 + gemma4.cpp
  / gemma4a.cpp (audio) / gemma4v.cpp (vision) + converter (text/MoE/vision/
  audio). It already implements SWA (LLAMA_SWA_TYPE_STANDARD), shared KV
  (n_kv_shared_layers → n_layer_kv_from_start reuse cb), PLE
  (per_layer_tok_embd), and interleaved-SWA KV (llama_kv_cache_iswa).

RAM / performance: llama.cpp is not optimal for Gemma out of the box

Gemma 4 is notorious for RAM blow-up in stock llama.cpp. Root causes + our levers
(several are config, not new kernels — "Gemma already does a lot of this"):

1. KV-cache layout (the ~40% fix) — landed upstream ~2026-04-05 (align KV
   memory layout to Gemma's MHA/grouping). Our merge-base is 2026-05-14, so we
   likely already have it; the M1 sync to upstream master confirms + grabs the
   rest.
2. ctx-checkpoints accumulation (#21690) — Gemma-only: server KV
   checkpoints grow unbounded (64 GB filled in 4–5 × 16K-prompt). Fix: bound
   ctx-checkpoints (1/0) + -np 1 in our fused defaults. Pure config.
3. Windowed SWA KV (swa_full=false) — llama_kv_cache_iswa already sizes
   SWA layers to n_swa (512/1024) + n_ubatch when swa_full=false. Most layers
   are SWA, so this is the dominant KV saving. Default swa_full=false (it's
   already plumbed through capacitor-llama/types.ts + the iOS shim).
4. PLE memory — per_layer_tok_embd is {n_embd_per_layer·n_layer, n_vocab}
   (~2.8B params for E2B). With mmap ON the OS pages it from disk (≈ LiteRT's
   "mmap embeddings, keep on disk until needed"). Never force --no-mmap for
   Gemma; on GPU backends, pin PLE to CPU/mmap rather than copying it to
   VRAM. (Watch Vulkan #18317: Vulkan can't run mmap=0.)
5. Prefix-KV reuse caveat (#21468) — Gemma's shared-KV layers log "cache reuse
   not supported" even with -fa --swa-full. So prefix reuse is partial on
   shared-KV layers — better than Qwen3.5 (which can't at all) but not free. Our
   stream_reset_keep work must account for this.

Leverage Google's C++ work, don't reinvent: LiteRT-LM already does
PLE-mmap + windowed-SWA + NPU optimally → use it as the in-process backend on
capable hardware. For llama.cpp, absorb upstream Gemma memory PRs and set
Gemma-aware defaults rather than porting our Qwen kernels blindly.

Where our kernels still add value (M6), re-scoped:

• TurboQuant low-precision weight quant — orthogonal to KV; applies to the
  dense FFN/attention weights. Keep.
• QJL/PolarQuant KV quant — now applied on top of the windowed SWA KV +
  the global-attention KV (where 256K context still hurts). Re-parameterize from
  uniform head_dim=128 to Gemma's dual dims (global ~256 + SWA). Validate it
  still wins after SWA+shared-KV already shrank the cache.
• Re-run the §8 8/8 kernel verify matrix per buildable backend for the new
  geometry.

Frozen contracts re-opened (owner-approved 2026-06-22)

1. Training base lock (model_registry.py, training/AGENTS.md) — Qwen → Gemma 4.
2. tokenizerFamily "qwen35"→"gemma4"; vocab 248,320 → 262,144 (memory_calc.py).
3. KV geometry + kernels — head_dim 128 uniform → Gemma dual dims.
4. Same-file MTP NextN head → Gemma separate drafter GGUF.
5. EOT <|im_end|> → <end_of_turn> (3 scorer files).
6. Abliteration Gated-DeltaNet surgery → dense surgery.
7. native/AGENTS.md §11 reinterpreted (single lib + single pipe).

eliza-1 v1 is base-not-fine-tuned (releaseState=base-v1) → v1 cutover is
swap base + re-optimize, not a retraining run.

Milestones (PR per milestone → develop)

• M1 — llama.cpp upstream sync + PR absorption. Merge ggml-org/master (602
  commits ahead of our base; we are 867 ahead with kernels/voice) into the fork;
  review + absorb all relevant Gemma/MTP/LiteRT/CoreML/AICore/KV/quant PRs (incl.
  open #21587 Gemma4 BPE SIGSEGV, #24590 Gemma4Assistant memory-fit, #21690
  ctx-checkpoints, #21468 cache-reuse). Build + verify Gemma 4 E2B/E4B
  text+vision+audio+MTP on CPU/CUDA.
• M2 — Code cutover Qwen→Gemma (registry, catalog/types, memory_calc, EOT,
  abliterate, AGENTS; remove Qwen; tier map E2B/E4B/12B/31B(/26B-A4B); Gemma-aware
  runtime defaults: swa_full=false, bounded ctx-checkpoints, mmap-on/PLE-on-CPU).
• M3 — Multi-backend FFI seam (backend abstraction + backend-selector).
• M4 — LiteRT-LM in-process backend (Android NPU delegate ladder).
• M5 — CoreML/MLX in-process backend (mac first, iOS later).
• M6 — Kernel re-optimization for Gemma geometry (TurboQuant weight-quant +
  QJL/Polar KV-quant on windowed/global KV) across CPU/CUDA/Vulkan-Mali/Metal/NPU
  • low-precision + long context; re-verify 8/8.
• M7 — Verification everywhere (web / desktop app / Pixel / bench harnesses +
  PR_EVIDENCE).

What's verifiable in-session vs needs hardware

• Here (Linux x64 + CUDA): llama.cpp merge + CPU/CUDA builds; Gemma 4
  text/vision/audio/MTP gen; code cutover; FFI seam; bench (llama-bench,
  e2e_loop_bench) on CPU/CUDA; web + desktop-app smoke.
• Needs Mac: Metal kernels, CoreML/MLX backends, iOS.
• Needs Pixel/Android device: Vulkan-Mali kernels, LiteRT NPU (Tensor/QNN),
  on-device tok/s + RSS. (Prior on-device work used adb/CDP on a Pixel 9a.)

Hardware-gated items are scoped + scaffolded here and marked for device
verification; nothing is claimed "verified" without the evidence.

Acceptance criteria

• Fork synced to upstream master; all relevant Gemma/MTP/LiteRT PRs absorbed or rejected-with-reason.
• Gemma 4 runs through libelizainference (text+vision+audio+MTP) on every buildable backend.
• Qwen fully removed from the shipped eliza-1 line.
• Multi-backend selection behind one FFI; LiteRT/MLX/CoreML in-process; AICore/Foundation opportunistic.
• Gemma-aware RAM defaults set; kernels re-optimized + 8/8 verified per buildable backend; low-precision quant validated.
• tok/s + RSS + first-token + MTP-acceptance captured per platform; faster-or-justified vs retired Qwen line.
• Verified on web + desktop app + on-device (as hardware allows; else honestly scoped).
• eliza-1 branding preserved (users never see "Qwen"/"Gemma").

[lalalune]

Status — 2026-06-22 (M0 + M2 landed)

Baseline proven: Gemma 4 E2B (Q8_0) runs through our llama.cpp fork unmodified — CPU pp64 56.6 t/s / tg32 5.84 t/s, build e4b3269fc. Real E2B geometry captured: 35 layers, hidden 1536, MQA (1 KV head), dual head dims 512 global / 256 SWA, 7 global + 28 SWA layers, 20 shared-KV layers, PLE 256, vocab 262144.

Key finding (reshapes M6): Gemma 4's KV is already minimal by construction (MQA + windowed-SWA + shared-KV), so the legacy QJL/PolarQuant KV-compression kernels are low-ROI and head-dim-incompatible (they're 128-dim; Gemma is 512/256). Gemma runs on stock q8_0 KV. TurboQuant weight-quant + separate-drafter MTP remain. RAM levers are config: swa_full=false, bounded ctx-checkpoints (#21690), mmap-on/PLE-on-CPU.

M2 — source-of-truth cutover (DONE, verified, pushed feat/gemma4-cutover):

• training: model_registry.py → Gemma 4 E2B/E4B/12B/31B; memory_calc vocab→262144; abliterate dense surgery; AGENTS/CLAUDE contracts.
• catalog/types/device-fit: tokenizerFamily qwen35→gemma4; same-file NextN MTP → separate official Gemma drafter; KV qjl1_256→stock q8_0.
• voice EOT: <|im_end|>→<end_of_turn> across 3 scorers + barrel.
• native §11 reinterpreted: single managed lib + single pipe (LiteRT/MLX/CoreML in-process compliant; AICore/Foundation opportunistic).
• Gates: shared + plugin-local-inference + benchmarks typecheck 0 errors; model-tiers test 11/11.

In flight: (M1) llama.cpp upstream-sync — fork is 602 commits behind upstream master (merge-base 5c0e94, 105-conflict trial merge); PR-review workflow producing the cherry-pick + conflict-resolution plan. Repo-wide residual-Qwen classification sweep running to complete the hard cutover.

HF (M8): hard cutover of elizaos/eliza-1 to Gemma bundles (no staging — not shipped, no downloaders yet).

[lalalune]

M1 progress + cut-list application — 2026-06-22

llama.cpp fork sync (M1):

• Shipped now: cherry-picked upstream #23981 (SWA checkpoints store only non-masked cells) → elizaOS/llama.cpp PR Awareness of Twitter bio and username #27. This is the structural Gemma KV-checkpoint RAM fix (#21690 OOM mitigation) on a path our FFI runs (common_prompt_checkpoint → state_write). Verified: clean cherry-pick, CPU rebuild green, Gemma 4 E2B still runs.
• Full 602-commit merge = its own gated effort. Expert review produced a resolution-ordered conflict guide; 3 hand-merges can cause wrong-output-no-crash (the ggml_gated_delta_net 3D-vs-4D op-contract, the KV-cache ctor arg collision, the fattn-common.cuh F16-buffer refactor on our head_dim=128 fused-attn). Key simplification: most of that conflict mass is Qwen-hybrid machinery we're abandoning (gated-delta-net, qwen35 recurrent discovery, same-file-NextN MTP) — Gemma 4 is dense, so we take-theirs aggressively and shed our Qwen divergence → easier merge + cleaner fork. Gemma-specific cherry-picks now in scope: #21587 (tokenizer SIGSEGV on long newline-free prompts — hits voice/transcript), #24590, #24014 (vision), #23247/#21433 (tool-call grammar).

M2 completion (residual-Qwen sweep): repo-wide classification found ~130 files with eliza-1-base Qwen refs; ~90% mechanical (training base-names/registry/HF-ids/tokens/vocab), load-bearing handful (manifest tokenizer constants, vision family qwen3-vl→gemma-vl + ChatML→Gemma template). Resolved the uncertain items from device data (Gemma head_dim 512/256 → stock KV stays; separate-drafter MTP → same-file-NextN is the retired path; 0_6b/0_8b dead). Applying now via fanned-out fix workflow with strict KEEP rules (Qwen3-ASR/OmniVoice/Embedding/turn-detector lineage + the banned-name guards stay).

HF authenticated (write access to elizaos/eliza-1 confirmed) → M8 unblocked.

[lalalune]

Impl-wave-1 landed (M6/M9/M10a/M10b + M2-tests) — 2026-06-22

Parallel disjoint implementation wave completed + verified green:

• M6 (kernels/quant): DECISION locked — KEEP geometry-agnostic GGUF weight-quant (TurboQuant); DROP QJL/PolarQuant/turbo3-KV for Gemma (head_dim=128-coupled; Gemma KV already minimal → stock q8_0). Kernels stay compiled but manifest-optional for Gemma tiers. Deleted dead Qwen-hybrid code: hybrid_cache.py, eagle3/ dir, serve_local hybrid branch, serve_vllm qwen3_next branches; retargeted the generic spec-decode manifest field.
• M9 (eliza-1-only): removed generic-GGUF machinery — generic-gguf-backend.ts deleted, RuntimeClass collapsed to fused-eliza1, arbitrary HF-search/assignment disabled, SlotAssignments → eliza-1 tiers only. (Local inference is bundle-locked to eliza-1 on desktop: surface real local model selection and a generic GGUF engine path #8808 stays closed.)
• M10a (cloud HF-proxy): new packages/cloud-api/v1/hf-proxy/[...path]/route.ts (auth + Range + streaming passthrough, cloud-side HF_TOKEN); packages/shared/src/local-inference/hf-proxy.ts resolveHfDownloadBase; catalog + downloader route HF through cloud when linked → no HF keys on device.
• M10b (memory): dynamic fit-to-RAM context selection in device-fit (per-token q8_0 KV rate, not just static minRam floor); memory-arbiter estimatedMb fix; eliza1_memory_inventory.mjs desktop bench harness.
• M2-tests: registry + memory_calc test partners rewritten to the 4-tier Gemma structure (32 pass).

Gates: packages/shared + plugin-local-inference + cloud-api typecheck 0 errors; runtime-class 6/6, device-fit 7/7, memory-arbiter + model_registry tests pass. Pushed to feat/gemma4-cutover (develop merged in).

Concurrent track: M3/M4/M5 (multi-backend FFI seam + LiteRT/MLX scaffolds) landed by a parallel actor in the llama.cpp fork (c849143c9). Next: M8 (build+publish Gemma E2B bundle — inputs downloaded), then M1 full upstream merge (gated), M7 verification.