perf(kokoro): Accelerate-BLAS hot loops (417x) + gemma4-assistant MTP converter/runtime#38
Closed
lalalune wants to merge 20 commits into
Closed
perf(kokoro): Accelerate-BLAS hot loops (417x) + gemma4-assistant MTP converter/runtime#38lalalune wants to merge 20 commits into
lalalune wants to merge 20 commits into
Conversation
…ly kokoro CLI The iOS XCFramework slice (build-llama-cpp-mtp.mjs, BUILD_SHARED_LIBS=OFF) builds `elizainference_static` by name and links libelizainference.a into the framework, but the target was never defined (only the SHARED `elizainference`), so ninja failed 'unknown target elizainference_static'. And tools/kokoro built its standalone CLI unconditionally with install(TARGETS kokoro-tts RUNTIME), which on iOS (executables are app bundles) fails configure with 'no BUNDLE DESTINATION'. - tools/omnivoice/CMakeLists.txt: add an elizainference_static STATIC target mirroring the SHARED one (same FFI+core sources, includes, OMNIVOICE_BUILD / ELIZA_ENABLE_VISION defines, and the if(TARGET kokoro_lib) Kokoro fold → ELIZA_ENABLE_KOKORO), OUTPUT_NAME elizainference → libelizainference.a; no -reexport_library (SHARED-only). - tools/kokoro/CMakeLists.txt: guard the host-only kokoro-tts CLI + its install off iOS. Verified: ios-arm64-metal-fused AND ios-arm64-simulator-metal-fused both build clean (320/320), libelizainference.a exports the four eliza_inference_kokoro_* symbols + tts/asr/eot, CAPABILITIES.json abiVersion=11 kokoro=kokoro-82m; the desktop SHARED build is unaffected (additive target). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rt (80→88) The streaming-LLM config struct gained `context_size` (int32 @ offset 80) for ABI v9 — `_llm_stream_open` now honors `cfg->context_size` (>0) instead of only the ELIZA_LLM_N_CTX env default. The TS marshaller (ffi-bindings.ts) was already emitting 88 bytes with context_size at offset 80, but the C-side ABI-guard `static_assert(sizeof(eliza_llm_stream_config_t) == 80)` was never bumped, so the fused `elizainference` target failed to compile. Bump it to 88 to match the real layout (8×int32 + 5×ptr, pointer-aligned). Validated: a host Metal build of `libelizainference` (ABI v12) loads + generates on the real google/gemma-4-E2B with context_size=4096 honored (83 tok/s, M4 Max). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d-correct sources Commit 412b848 ('fix(metal): correct TBQ3_0/TBQ4_0 attention score kernels') regressed kernel_turbo3_dot/kernel_turbo4_dot in eliza-shipped/: standalone metal_verify drops to 0/8 (outputs 20-170x off) and the built-fork Metal graph-dispatch smoke fails GGML_OP_ATTN_SCORE_TBQ/turbo3+turbo4. The metal-verify gate only tested native/metal/*.metal (the verify-harness copy), not the eliza-shipped/ copies the runtime embeds, so the regression shipped unguarded. Restore eliza-shipped/turbo3.metal + turbo4.metal from the verified native/metal copies (metal_verify 8/8). After the fix: dispatch_smoke PASS 9/9 routes; all shipped kernels pass parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ther seam - LLM_ARCH_GEMMA4_ASSISTANT enum + arch name + nextn/masked tensor enums/names/infos - hparams n_embd_inp_impl + n_embd_inp() override - model fields nextn_proj_pre/post + models.h struct decl - llm_graph_result t_h_nextn + get_h_nextn - src/models/gemma4-assistant.cpp (ported from upstream, fork naming) - ctx_other on cparams + llama_context_params + llama_get_ctx_other - llama_memory_params.mem_other + layer_share_cb Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-type - llama_kv_cache / llama_kv_cache_iswa ctors take mem_other + share - per-layer share() maps drafter layers to sibling target KV layers - all existing callers pass nullptr (additive, behavior unchanged) - model factory dispatch + NEOX rope-type for gemma4-assistant - create_memory builds share lambda + mem_other for gemma4-assistant Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
server-context sets cparams.ctx_other = ctx_tgt on both MTP draft-context branches so the gemma4-assistant drafter can read the target's token embeddings and share its KV cache. Harmless for other draft arches (only that arch reads it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…edup Three correctness fixes that take draft acceptance from ~2% to ~41%: 1. MTP runtime row width = n_embd_out (backbone 1536), not the drafter's internal n_embd (256); assert it matches the target hidden width. 2. pre-norm extraction (context.cpp) + get_embeddings_pre_norm_ith + embd_pre_norm buffer now sized/strided by n_embd_out, so the n_embd_out-wide nextn hidden survives the round trip. 3. target gemma4 graph now exposes its POST-final-norm hidden via t_h_pre_norm (the LM-head input feature the drafter consumes), matching upstream's t_h_nextn; gemma4-assistant routes its nextn hidden through the same seam. Result on M4 Max Metal: 133.8 t/s with draft-mtp vs 109.5 t/s baseline (1.22x). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
embd_pre_norm.size now derives from n_embd_out; the n_embd local is dead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ant)
The fused engine builds its MTP draft context (eliza-inference-ffi.cpp) without
ctx_other, so a separate-drafter MTP arch that reaches into the target context
(gemma4-assistant: target tok_embd + hidden) hard-failed in the llama-context
ctor ("Gemma4Assistant requires ctx_other to be set"). server-context.cpp set it;
the fused FFI path did not. Set cp.ctx_other = e->ctx_tgt before init (inert for
same-file MTP archs that don't consult ctx_other).
Validated via the fused FFI on M4 Max Metal: loads the amaranus gemma4-assistant
drafter + runs MTP — drafted 28 / accepted 12 (43%), coherent output, 112 tok/s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…9460) The pyannote-segmentation-3.0 GGUF carries ONNX-exported LSTM weights, whose gate blocks are laid out IOFC (input, output, forget, cell). voice_diarizer.c's lstm_run_dir read them in PyTorch IFGO order (gi@0,gf@1,gg@2,go@3), scrambling forget/cell/output -> garbage segmentation -> spurious 3rd speaker + 100+ fragments on clean 2-speaker audio (#9460 over-detection). Reorder to IOFC (gi@0,go@1,gf@2,gg@3). Verified on Apple M-series with the local real-acoustic harness (freeman.wav + Kokoro, 2-speaker GT): IFGO -> 121 segments / 3 speakers / DER ~57%; IOFC -> 2 segments / 2 speakers / DER ~32%. Mirrors develop's landed fix a359942d30 onto the gemma4-assistant fork branch (submodule 0864259), which still carried the pre-fix order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…9588)
The fused-lib loader required unprefixed dev tensor names (bert.embd.tok.weight,
pred.F0_proj.weight, dec.gen.conv_post.weight) while the published elizaOS Kokoro
bundles use the kokoro.* namespace and mainline llama.cpp names
(kokoro.bert.layer.attn_q.weight). The loader, the shipped GGUF, and the in-tree
converter all disagreed, so kokoro_init_from_file failed its weight sanity check
for every available GGUF and the engine silently fell through to OmniVoice/stub.
Two root causes, two fixes:
1. Tensor-name mismatch. Centralize the accepted name variants in
kokoro-tensor-names.h (published + mainline + legacy dev) and look them up via
require_tensor_any(). Required tensors are now a HARD load error instead of the
old "non-fatal during J2 — treat absent tensors as zero" path, which produced
shape-correct but acoustically degraded (noise) output and masked this bug.
2. All-F32 GGUFs load but synthesize noise. The converter now emits weight
matrices / conv kernels (ndim >= 2) as F16 and keeps biases/norms F32, matching
the dtype layout the fused forward pass expects. The stub emitter also writes
kokoro.gen.conv_post.{weight,bias} so it passes the new required-tensor check.
Adds test_kokoro_tensor_names.cpp (LLAMA_BUILD_TESTS-gated) asserting the alias
picker resolves published, mainline, and legacy schemas and returns null when a
tensor is genuinely absent. Closes the loader half of #9588; regenerating +
republishing the bundle GGUF with this converter is the remaining ops step.
…peak G2P (#9588)
Kokoro TTS never actually spoke: `kokoro_synthesize` ran a placeholder
spectrogram (envelope x 1/f timbre x random phase) and discarded the model,
and the scalar predictor read F16/quantized GGUF tensors as raw F32 (garbage),
collapsing BERT to a constant -> a fixed-pitch buzz. This implements the real
forward pass end-to-end, validated against the PyTorch reference and verified
intelligible by whisper on the SHIPPED Q4_K_M bundle (no GGUF republish needed).
Root causes fixed:
- DTYPE: kokoro_load_model now dequantizes every non-F32 tensor to a parallel
all-F32 context (ggml to_float traits: F16 + Q5_0/Q4_K/Q6_K). The
predictor/decoder read F32 only. This alone un-breaks the (already-correct)
prosody predictor — BERT now matches the reference to 3 decimals.
- SYNTHESIS WAS A STUB: implement the iSTFTNet decoder.
- kokoro-decoder-front.h: Decoder.forward up to the generator (F0/N conv,
encode, asr_res, 4 AdainResBlk1d decode blocks). Validated corr=1.0/stage.
- kokoro-generator.{h,cpp}: the Generator — harmonic source (SineGen+Source
ModuleHnNSF from F0), center=True STFT, 2 ConvTranspose upsamples, 6 Snake
AdaINResBlock1, noise convs/res, conv_post, center=True iSTFT. corr 0.993
vs reference; whisper transcribes the output.
- kokoro-decoder.{h,cpp}: binds the model's F32 tensors into the front +
generator weight structs and runs them; wired into kokoro_synthesize
(predictor -> transpose asr -> decoder), variable-length T.
- G2P: kokoro-phonemes now drives libespeak-ng (en-us IPA -> Kokoro vocab ids,
reproducing the reference token sequence) when linked; ASCII fallback only
when espeak is absent. CMake auto-detects libespeak-ng.
Validation (Apple M-series): full text->speech via kokoro-tts on BOTH the
all-F32 regen GGUF and the published Q4_K_M bundle, whisper-intelligible across
phrases ("Hello, this is a native Kokoro voice test.", "The quick brown fox
...", "Open the pod bay doors please."). Per-component standalone tests
(decoder front, Snake resblock, harmonic source+STFT, iSTFT) corr 1.0 vs the
PyTorch reference. New unit test test_kokoro_g2p_espeak; kokoro-stage-dump /
kokoro-decoder-test dev harnesses. The published bundle GGUF is unchanged.
…TP drafter) Register the gemma4-assistant arch in gguf-py (MODEL_ARCH.GEMMA4_ASSISTANT, NEXTN_PROJ_PRE/POST tensor names + HF pre_projection/post_projection mapping) and add Gemma4AssistantModel to convert_hf_to_gguf.py. Writes embedding_length_out = backbone_hidden_size (the target hidden width the pre/post projections run at) and nextn_predict_layers = 1 (the MTP-context gate), drops the HF masked_embedding lookup tables the llama.cpp graph does not use, and inherits the Gemma4 swa-pattern/shared-kv/dual-head-dim/ proportional-rope handling. Verified against google/gemma-4-E2B-it-assistant (49 tensors, f16, embd 256 -> out 1536). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ugh Accelerate BLAS — 417x synth speedup on M4 Max (RTF 64x -> 0.15) Root cause of the 302s/4.7s-audio synth (#9033 kokoro perf leg): the StyleTTS-2 decoder + iSTFTNet generator never enter ggml — kokoro-layers.h is a single-threaded scalar port, so nothing was ever dispatched to Metal. The generator's ~85 GMACs of conv work ran as branchy -O3 scalar loops (261s, 87% of synth). Fix: on __APPLE__, kokoro-layers.h now routes the four hot primitives through Accelerate (AMX): - conv1d_forward: im2col + one cblas_sgemm (weights [Cout,Cin,K] are already a row-major [Cout,Cin*K] GEMM operand; stride/dilation/zero-pad baked into the column matrix) - convtranspose1d_forward: one cblas_sgemm (tmp = W^T x) + col2im scatter-add - linear_forward: cblas_sgemv - lstm_cell_step gates: two cblas_sgemv (predictor 24.2s -> 0.11s) Scalar loops stay as the non-Apple fallback + readable reference. CMake links Accelerate.framework into kokoro_lib (PUBLIC) on APPLE; verified propagated to kokoro-tts and libelizainference.dylib. Also lands the kokoro-profile phase-timing instrumentation (KOKORO_PROFILE=1) used to attribute the baseline: kokoro-profile.h RAII timer + marks in kokoro.cpp / kokoro-decoder.cpp / kokoro-generator.cpp. Measured (M4 Max, build-desktop-metal Release, same phrase/model/voice): predictor 24,188 -> 113 ms; decoder front 16,288 -> 15 ms; generator 261,220 -> 595 ms; synth total 301,697 -> 723 ms (RTF 64x slower-than-RT -> 0.15, ~6.5x faster than RT). Audio unchanged: identical 112,800 samples, corr 0.99959, RMS equal, identical whisper transcript. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…he four hot layer primitives Android/Windows/Linux previously ran the single-threaded scalar loops in kokoro-layers.h (261 s for the iSTFTNet generator on a 4.7 s utterance). The Accelerate BLAS speedup (prev commit) is __APPLE__-only. - conv1d_forward / convtranspose1d_forward / linear_forward / lstm_cell_step gates now have a KOKORO_USE_PORTABLE_FAST branch: a small internal std::thread pool (min(hw_concurrency,16), override KOKORO_NUM_THREADS) parallelizes over output channels / gate rows, and the innermost MACs run branch-free AXPY/dot kernels — NEON vfmaq_f32 on aarch64, auto-vectorizable scalar elsewhere. Valid-range hoisting removes the per-element bounds checks of the reference loop. - Pure scalar loops kept verbatim as the reference/ultimate fallback (-DKOKORO_FORCE_SCALAR); -DKOKORO_NO_ACCELERATE forces the portable path on Apple (used by the parity test). - tests/test_kokoro_layers_portable: 23 parity checks fast-vs-scalar on random input (max |Δ| 1.9e-5 < 1e-4, incl. stride/dilation/OOB-tap edges) + microbench on real generator shapes. M4 Max (16 threads, NEON): resblock convs 165–173x, upsample transpose convs 15–25x, predictor-sized LSTM 4.1x vs scalar. - CMake: link Threads::Threads on non-Apple; wire the new test. Cross-checked with NDK r29 clang for aarch64-linux-android26 and x86_64-linux-android26 (-Wall -Wextra clean; 18 fmla.4s in the aarch64 object). Apple Accelerate path unchanged and rebuilt green (build-desktop-metal kokoro-tts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8 tasks
Member
Author
|
Fast-forwarded this branch
Correctness executed on real arm64: 23/23 parity checks pass (max |Δ| 1.9e-5 vs scalar reference), NDK r29 cross-compile clean (18 |
lalalune
pushed a commit
to elizaOS/eliza
that referenced
this pull request
Jul 2, 2026
…e non-Apple kokoro fast path Adds Android/Windows kokoro acceleration on top of the Apple Accelerate path already on develop (114eee08e). The non-Apple branch of the four hot vocoder primitives (Conv1d/ConvTranspose1d/Linear/LSTM gates) now uses an internal std::thread pool + NEON vfmaq_f32 on aarch64 (auto-vectorizable scalar on x86); pure scalar loop retained as reference. Apple path unchanged. Verified on real arm64: 23/23 parity checks vs scalar (max |Δ| 1.9e-5), NDK r29 cross-compile clean (18 fmla.4s in aarch64 object), microbench up to 107x on conv1d. Fork: elizaOS/llama.cpp#38 (perf/kokoro-accelerate-blas @ 2bdcef890). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lalalune
added a commit
to elizaOS/eliza
that referenced
this pull request
Jul 2, 2026
…e non-Apple kokoro fast path (#11584) Adds Android/Windows kokoro acceleration on top of the Apple Accelerate path already on develop (114eee08e). The non-Apple branch of the four hot vocoder primitives (Conv1d/ConvTranspose1d/Linear/LSTM gates) now uses an internal std::thread pool + NEON vfmaq_f32 on aarch64 (auto-vectorizable scalar on x86); pure scalar loop retained as reference. Apple path unchanged. Verified on real arm64: 23/23 parity checks vs scalar (max |Δ| 1.9e-5), NDK r29 cross-compile clean (18 fmla.4s in aarch64 object), microbench up to 107x on conv1d. Fork: elizaOS/llama.cpp#38 (perf/kokoro-accelerate-blas @ 2bdcef890). Co-authored-by: Shaw <shawgotbags@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(omnivoice): reject explicit diarizer gate-order skew
Member
Author
|
Superseded by #42. Its substantive commits are already ancestors of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fork branch
perf/kokoro-accelerate-blas@114eee08e— the commit the Milady/eliza monorepo submodule gitlink now points at (see companion monorepo PR "feat(local-inference): Gemma-4 cutover finish"). Two workstreams:1. Kokoro TTS: Accelerate-BLAS hot loops (417x synth speedup)
perf(kokoro): route Conv1d / ConvTranspose1d / Linear / LSTM hot loops through Accelerate BLAS (cblas_sgemm/cblas_sgemv) on Apple platforms..github/issue-evidence/kokoro-metal-perf/.dur_projalias.2. gemma4-assistant: HF->GGUF converter + MTP drafter runtime
feat(convert):Gemma4AssistantForCausalLM->gemma4-assistantGGUF (native MTP drafter for eliza-1, 78 MB) — commitae0a1eed2, contained in this branch.ctx_other=ctx_tgt) for draft-MTP contexts, corrected MTP hidden-state feedback, fused-FFI ABI v9context_size.fix(build): unbreak the iOS fused slice (static FFI target + host-only kokoro CLI).fix(metal): revert broken TBQ3_0/TBQ4_0 attn-score kernels to verified-correct sources.fix(voice-diarizer): pyannote-3 LSTM gate order (ONNX IOFC).Why merge to main
The monorepo
.gitmodulestracks this fork'smain; the submodule bump in the monorepo PR pins114eee08edirectly, but the gitlink needs an upstream home on the default branch so futuremain-tracking updates don't orphan it.Evidence
.github/issue-evidence/kokoro-metal-perf/(before/after WAV + profiles),.github/issue-evidence/gemma4-assistant-mtp/(drafter GGUF metadata, draft-acceptance bench, fused FFI run),.github/issue-evidence/vision-gemma4-describe/.🤖 Generated with Claude Code