Reconcile main with the develop-pinned lineage (perf/kokoro-accelerate-blas + #11612 Metal guards)#42
Merged
Conversation
…ly kokoro CLI The iOS XCFramework slice (build-llama-cpp-mtp.mjs, BUILD_SHARED_LIBS=OFF) builds `elizainference_static` by name and links libelizainference.a into the framework, but the target was never defined (only the SHARED `elizainference`), so ninja failed 'unknown target elizainference_static'. And tools/kokoro built its standalone CLI unconditionally with install(TARGETS kokoro-tts RUNTIME), which on iOS (executables are app bundles) fails configure with 'no BUNDLE DESTINATION'. - tools/omnivoice/CMakeLists.txt: add an elizainference_static STATIC target mirroring the SHARED one (same FFI+core sources, includes, OMNIVOICE_BUILD / ELIZA_ENABLE_VISION defines, and the if(TARGET kokoro_lib) Kokoro fold → ELIZA_ENABLE_KOKORO), OUTPUT_NAME elizainference → libelizainference.a; no -reexport_library (SHARED-only). - tools/kokoro/CMakeLists.txt: guard the host-only kokoro-tts CLI + its install off iOS. Verified: ios-arm64-metal-fused AND ios-arm64-simulator-metal-fused both build clean (320/320), libelizainference.a exports the four eliza_inference_kokoro_* symbols + tts/asr/eot, CAPABILITIES.json abiVersion=11 kokoro=kokoro-82m; the desktop SHARED build is unaffected (additive target). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rt (80→88) The streaming-LLM config struct gained `context_size` (int32 @ offset 80) for ABI v9 — `_llm_stream_open` now honors `cfg->context_size` (>0) instead of only the ELIZA_LLM_N_CTX env default. The TS marshaller (ffi-bindings.ts) was already emitting 88 bytes with context_size at offset 80, but the C-side ABI-guard `static_assert(sizeof(eliza_llm_stream_config_t) == 80)` was never bumped, so the fused `elizainference` target failed to compile. Bump it to 88 to match the real layout (8×int32 + 5×ptr, pointer-aligned). Validated: a host Metal build of `libelizainference` (ABI v12) loads + generates on the real google/gemma-4-E2B with context_size=4096 honored (83 tok/s, M4 Max). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d-correct sources Commit 412b848 ('fix(metal): correct TBQ3_0/TBQ4_0 attention score kernels') regressed kernel_turbo3_dot/kernel_turbo4_dot in eliza-shipped/: standalone metal_verify drops to 0/8 (outputs 20-170x off) and the built-fork Metal graph-dispatch smoke fails GGML_OP_ATTN_SCORE_TBQ/turbo3+turbo4. The metal-verify gate only tested native/metal/*.metal (the verify-harness copy), not the eliza-shipped/ copies the runtime embeds, so the regression shipped unguarded. Restore eliza-shipped/turbo3.metal + turbo4.metal from the verified native/metal copies (metal_verify 8/8). After the fix: dispatch_smoke PASS 9/9 routes; all shipped kernels pass parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ther seam - LLM_ARCH_GEMMA4_ASSISTANT enum + arch name + nextn/masked tensor enums/names/infos - hparams n_embd_inp_impl + n_embd_inp() override - model fields nextn_proj_pre/post + models.h struct decl - llm_graph_result t_h_nextn + get_h_nextn - src/models/gemma4-assistant.cpp (ported from upstream, fork naming) - ctx_other on cparams + llama_context_params + llama_get_ctx_other - llama_memory_params.mem_other + layer_share_cb Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-type - llama_kv_cache / llama_kv_cache_iswa ctors take mem_other + share - per-layer share() maps drafter layers to sibling target KV layers - all existing callers pass nullptr (additive, behavior unchanged) - model factory dispatch + NEOX rope-type for gemma4-assistant - create_memory builds share lambda + mem_other for gemma4-assistant Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
server-context sets cparams.ctx_other = ctx_tgt on both MTP draft-context branches so the gemma4-assistant drafter can read the target's token embeddings and share its KV cache. Harmless for other draft arches (only that arch reads it). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…edup Three correctness fixes that take draft acceptance from ~2% to ~41%: 1. MTP runtime row width = n_embd_out (backbone 1536), not the drafter's internal n_embd (256); assert it matches the target hidden width. 2. pre-norm extraction (context.cpp) + get_embeddings_pre_norm_ith + embd_pre_norm buffer now sized/strided by n_embd_out, so the n_embd_out-wide nextn hidden survives the round trip. 3. target gemma4 graph now exposes its POST-final-norm hidden via t_h_pre_norm (the LM-head input feature the drafter consumes), matching upstream's t_h_nextn; gemma4-assistant routes its nextn hidden through the same seam. Result on M4 Max Metal: 133.8 t/s with draft-mtp vs 109.5 t/s baseline (1.22x). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
embd_pre_norm.size now derives from n_embd_out; the n_embd local is dead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ant)
The fused engine builds its MTP draft context (eliza-inference-ffi.cpp) without
ctx_other, so a separate-drafter MTP arch that reaches into the target context
(gemma4-assistant: target tok_embd + hidden) hard-failed in the llama-context
ctor ("Gemma4Assistant requires ctx_other to be set"). server-context.cpp set it;
the fused FFI path did not. Set cp.ctx_other = e->ctx_tgt before init (inert for
same-file MTP archs that don't consult ctx_other).
Validated via the fused FFI on M4 Max Metal: loads the amaranus gemma4-assistant
drafter + runs MTP — drafted 28 / accepted 12 (43%), coherent output, 112 tok/s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…9460) The pyannote-segmentation-3.0 GGUF carries ONNX-exported LSTM weights, whose gate blocks are laid out IOFC (input, output, forget, cell). voice_diarizer.c's lstm_run_dir read them in PyTorch IFGO order (gi@0,gf@1,gg@2,go@3), scrambling forget/cell/output -> garbage segmentation -> spurious 3rd speaker + 100+ fragments on clean 2-speaker audio (#9460 over-detection). Reorder to IOFC (gi@0,go@1,gf@2,gg@3). Verified on Apple M-series with the local real-acoustic harness (freeman.wav + Kokoro, 2-speaker GT): IFGO -> 121 segments / 3 speakers / DER ~57%; IOFC -> 2 segments / 2 speakers / DER ~32%. Mirrors develop's landed fix a359942d30 onto the gemma4-assistant fork branch (submodule 0864259), which still carried the pre-fix order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…9588)
The fused-lib loader required unprefixed dev tensor names (bert.embd.tok.weight,
pred.F0_proj.weight, dec.gen.conv_post.weight) while the published elizaOS Kokoro
bundles use the kokoro.* namespace and mainline llama.cpp names
(kokoro.bert.layer.attn_q.weight). The loader, the shipped GGUF, and the in-tree
converter all disagreed, so kokoro_init_from_file failed its weight sanity check
for every available GGUF and the engine silently fell through to OmniVoice/stub.
Two root causes, two fixes:
1. Tensor-name mismatch. Centralize the accepted name variants in
kokoro-tensor-names.h (published + mainline + legacy dev) and look them up via
require_tensor_any(). Required tensors are now a HARD load error instead of the
old "non-fatal during J2 — treat absent tensors as zero" path, which produced
shape-correct but acoustically degraded (noise) output and masked this bug.
2. All-F32 GGUFs load but synthesize noise. The converter now emits weight
matrices / conv kernels (ndim >= 2) as F16 and keeps biases/norms F32, matching
the dtype layout the fused forward pass expects. The stub emitter also writes
kokoro.gen.conv_post.{weight,bias} so it passes the new required-tensor check.
Adds test_kokoro_tensor_names.cpp (LLAMA_BUILD_TESTS-gated) asserting the alias
picker resolves published, mainline, and legacy schemas and returns null when a
tensor is genuinely absent. Closes the loader half of #9588; regenerating +
republishing the bundle GGUF with this converter is the remaining ops step.
…peak G2P (#9588)
Kokoro TTS never actually spoke: `kokoro_synthesize` ran a placeholder
spectrogram (envelope x 1/f timbre x random phase) and discarded the model,
and the scalar predictor read F16/quantized GGUF tensors as raw F32 (garbage),
collapsing BERT to a constant -> a fixed-pitch buzz. This implements the real
forward pass end-to-end, validated against the PyTorch reference and verified
intelligible by whisper on the SHIPPED Q4_K_M bundle (no GGUF republish needed).
Root causes fixed:
- DTYPE: kokoro_load_model now dequantizes every non-F32 tensor to a parallel
all-F32 context (ggml to_float traits: F16 + Q5_0/Q4_K/Q6_K). The
predictor/decoder read F32 only. This alone un-breaks the (already-correct)
prosody predictor — BERT now matches the reference to 3 decimals.
- SYNTHESIS WAS A STUB: implement the iSTFTNet decoder.
- kokoro-decoder-front.h: Decoder.forward up to the generator (F0/N conv,
encode, asr_res, 4 AdainResBlk1d decode blocks). Validated corr=1.0/stage.
- kokoro-generator.{h,cpp}: the Generator — harmonic source (SineGen+Source
ModuleHnNSF from F0), center=True STFT, 2 ConvTranspose upsamples, 6 Snake
AdaINResBlock1, noise convs/res, conv_post, center=True iSTFT. corr 0.993
vs reference; whisper transcribes the output.
- kokoro-decoder.{h,cpp}: binds the model's F32 tensors into the front +
generator weight structs and runs them; wired into kokoro_synthesize
(predictor -> transpose asr -> decoder), variable-length T.
- G2P: kokoro-phonemes now drives libespeak-ng (en-us IPA -> Kokoro vocab ids,
reproducing the reference token sequence) when linked; ASCII fallback only
when espeak is absent. CMake auto-detects libespeak-ng.
Validation (Apple M-series): full text->speech via kokoro-tts on BOTH the
all-F32 regen GGUF and the published Q4_K_M bundle, whisper-intelligible across
phrases ("Hello, this is a native Kokoro voice test.", "The quick brown fox
...", "Open the pod bay doors please."). Per-component standalone tests
(decoder front, Snake resblock, harmonic source+STFT, iSTFT) corr 1.0 vs the
PyTorch reference. New unit test test_kokoro_g2p_espeak; kokoro-stage-dump /
kokoro-decoder-test dev harnesses. The published bundle GGUF is unchanged.
…TP drafter) Register the gemma4-assistant arch in gguf-py (MODEL_ARCH.GEMMA4_ASSISTANT, NEXTN_PROJ_PRE/POST tensor names + HF pre_projection/post_projection mapping) and add Gemma4AssistantModel to convert_hf_to_gguf.py. Writes embedding_length_out = backbone_hidden_size (the target hidden width the pre/post projections run at) and nextn_predict_layers = 1 (the MTP-context gate), drops the HF masked_embedding lookup tables the llama.cpp graph does not use, and inherits the Gemma4 swa-pattern/shared-kv/dual-head-dim/ proportional-rope handling. Verified against google/gemma-4-E2B-it-assistant (49 tensors, f16, embd 256 -> out 1536). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ugh Accelerate BLAS — 417x synth speedup on M4 Max (RTF 64x -> 0.15) Root cause of the 302s/4.7s-audio synth (#9033 kokoro perf leg): the StyleTTS-2 decoder + iSTFTNet generator never enter ggml — kokoro-layers.h is a single-threaded scalar port, so nothing was ever dispatched to Metal. The generator's ~85 GMACs of conv work ran as branchy -O3 scalar loops (261s, 87% of synth). Fix: on __APPLE__, kokoro-layers.h now routes the four hot primitives through Accelerate (AMX): - conv1d_forward: im2col + one cblas_sgemm (weights [Cout,Cin,K] are already a row-major [Cout,Cin*K] GEMM operand; stride/dilation/zero-pad baked into the column matrix) - convtranspose1d_forward: one cblas_sgemm (tmp = W^T x) + col2im scatter-add - linear_forward: cblas_sgemv - lstm_cell_step gates: two cblas_sgemv (predictor 24.2s -> 0.11s) Scalar loops stay as the non-Apple fallback + readable reference. CMake links Accelerate.framework into kokoro_lib (PUBLIC) on APPLE; verified propagated to kokoro-tts and libelizainference.dylib. Also lands the kokoro-profile phase-timing instrumentation (KOKORO_PROFILE=1) used to attribute the baseline: kokoro-profile.h RAII timer + marks in kokoro.cpp / kokoro-decoder.cpp / kokoro-generator.cpp. Measured (M4 Max, build-desktop-metal Release, same phrase/model/voice): predictor 24,188 -> 113 ms; decoder front 16,288 -> 15 ms; generator 261,220 -> 595 ms; synth total 301,697 -> 723 ms (RTF 64x slower-than-RT -> 0.15, ~6.5x faster than RT). Audio unchanged: identical 112,800 samples, corr 0.99959, RMS equal, identical whisper transcript. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he four hot layer primitives Android/Windows/Linux previously ran the single-threaded scalar loops in kokoro-layers.h (261 s for the iSTFTNet generator on a 4.7 s utterance). The Accelerate BLAS speedup (prev commit) is __APPLE__-only. - conv1d_forward / convtranspose1d_forward / linear_forward / lstm_cell_step gates now have a KOKORO_USE_PORTABLE_FAST branch: a small internal std::thread pool (min(hw_concurrency,16), override KOKORO_NUM_THREADS) parallelizes over output channels / gate rows, and the innermost MACs run branch-free AXPY/dot kernels — NEON vfmaq_f32 on aarch64, auto-vectorizable scalar elsewhere. Valid-range hoisting removes the per-element bounds checks of the reference loop. - Pure scalar loops kept verbatim as the reference/ultimate fallback (-DKOKORO_FORCE_SCALAR); -DKOKORO_NO_ACCELERATE forces the portable path on Apple (used by the parity test). - tests/test_kokoro_layers_portable: 23 parity checks fast-vs-scalar on random input (max |Δ| 1.9e-5 < 1e-4, incl. stride/dilation/OOB-tap edges) + microbench on real generator shapes. M4 Max (16 threads, NEON): resblock convs 165–173x, upsample transpose convs 15–25x, predictor-sized LSTM 4.1x vs scalar. - CMake: link Threads::Threads on non-Apple; wire the new test. Cross-checked with NDK r29 clang for aarch64-linux-android26 and x86_64-linux-android26 (-Wall -Wextra clean; 18 fmla.4s in the aarch64 object). Apple Accelerate path unchanged and rebuilt green (build-desktop-metal kokoro-tts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ash (#11612) A18 devices can fail to compile individual Metal compute pipelines (e.g. mul_mat variants) at runtime. Previously that either segfaulted (NULL pipeline deref) or GGML_ABORT'd via the defensive guard, crashing the whole app in a loop. Now a nil pipeline: - latches encode_failed on the command encoder (all subsequent encode calls on that encoder become no-ops, so nothing dispatches with unset/stale pipeline state), - makes ggml_metal_op_encode return 0 with the failing op logged, - puts the backend into the has_error state so ggml_metal_graph_compute returns GGML_STATUS_FAILED, - which surfaces as a llama_decode error the host can catch and route to a clean error / cloud fallback instead of a crash-loop. GGML_METAL_ABORT_ON_NIL_PIPELINE=1 restores the legacy hard-abort for debugging. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… bf16 kernels (#11612) The iOS embedded metallib was compiled at -std=ios-metal2.4 (__METAL_VERSION__ = 240), so ggml-metal.metal's guard '#if __METAL_VERSION__ < 310 && defined(GGML_METAL_HAS_BF16)' silently dropped the entire bf16 kernel family from default.metallib, while the runtime still set props.has_bfloat from the GPU family probe (Apple6+/ Metal3). On A18 (iPhone 16 Pro Max) loading eliza-1-2b (1 bf16 tensor) then selected kernel_mul_mm_bf16_f32, got MTLLibraryError Code=5 'Function not found', and every decode graph failed. Fix: after the Metal library is created, probe it for kernel_mul_mm_bf16_f32; when the library lacks the bf16 kernels, downgrade props.has_bfloat so supports_op routes bf16 ops to the CPU backend instead of failing generation. Also document in the metallib CMake block that GGML_METAL_STD must be metal3.1+ for bf16 kernels to be emitted (the build-side fix bumps the iOS default to metal3.1 in packages/app-core/scripts/build-llama-cpp-mtp.mjs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2 Metal guards) into main Reconciles fork main with the commit chain the elizaOS develop submodule pointer actually ships (elizaOS/eliza#11386, #9033 M1): ba598f5 feat(kokoro): real StyleTTS-2/iSTFTNet decoder (#9588) ae0a1ee feat(convert): gemma4-assistant GGUF MTP drafter converter 114eee0 perf(kokoro): Accelerate-BLAS hot loops (417x on M4 Max) 2bdcef8 perf(kokoro): portable non-Apple fast path (threads + NEON) c3b9fa6 fix(metal): recoverable nil compute pipeline (#11612) 58c0391 fix(metal): gate has_bfloat on loaded library (#11612) plus the gemma4-assistant arch stack, ABI v9 context_size fix, TBQ3_0/TBQ4_0 attn-score kernel revert, and the pyannote-3 IOFC diarizer gate fix already on that lineage. Conflict resolution (tools/kokoro/*): favored the pinned lineage (it is what ships) — main's #36/#37 kokoro re-lands are older snapshots of the same work. Main-only fixes preserved on top: kokoro_lib POSITION_INDEPENDENT_CODE, MACOSX_BUNDLE OFF CLI harnesses, and the GGML_BACKEND_DL-safe registry backend init in kokoro_load_model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…fail-closed epoch guard The vendored voice_diarizer.c still read pyannote LSTM gates in raw ONNX IOFC order (the original C-side #9460 fix) while the package-side converter moved the reorder converter-side and now bakes IFGO. Any artifact x reader packing skew silently scrambles the gates into the DER 1.000 over-segmentation signature (elizaOS/eliza#11377). - flip the LSTM gate unpack to I,F,G,O — matches packages/native/plugins/voice-classifier-cpp voice_diarizer.c on elizaOS/eliza develop (post-#11569) - port the #11569 fail-closed guard: require voice_diarizer.converter_epoch >= 2 and lstm_gate_order == "IFGO"; epoch-less legacy artifacts (the previously published IOFC bake) and explicit non-IFGO artifacts are rejected loudly before tensor load - extend the #40 metadata contract test to the fail-closed contract: epoch-2 IFGO clears the guard (reaches missing-tensor), explicit IOFC, epoch-less, and epoch-1 artifacts are all rejected with the exact guard message (stderr captured); metadata-only GGUFs padded to the 32-byte data alignment so the zero-tensor mapper accepts them Requires the coordinated epoch-2 IFGO artifact republish on HF (elizaos/eliza-1 voice/diarizer/) before this ships in a fused lib pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
IFGO reader sync) into the main reconciliation develop's submodule gitlink advanced to dda200a while this reconciliation was in flight; fold the two diarizer commits in so main contains the full shipped lineage (elizaOS/eliza#11386). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the "main lags the shipped pointer" gap from elizaOS/eliza#11386 (#9033 M1 follow-up).
What
Merges the commit chain that the elizaOS
developsubmodule pointer (plugins/plugin-local-inference/native/llama.cpp@58c0391eb) actually ships into forkmain:plus everything under it already on that lineage: the
gemma4-assistantMTP arch stack, the fused-FFI ABI v9context_sizefix, the TBQ3_0/TBQ4_0 attn-score kernel revert, and the pyannote-3 IOFC diarizer gate fix.After this merge,
maincontains everythingdeveloppins and everythingmainhad that the lineage lacked (upstream #23981 SWA checkpoints via #27, LiteRT per-op backend seam #35, vision streaming ABI v13 #33, streaming-LLM FFI seam, kokoro CI-matrix fixes #28).Conflict resolution
Conflicts were confined to
tools/kokoro/*: main's #36/#37 kokoro commits were older re-lands of the same lineage work. Resolved favoring the pinned lineage (it is what ships), keeping the three main-only fixes on top:kokoro_libPOSITION_INDEPENDENT_CODE ON(OpenVINO Linux CI shared build)MACOSX_BUNDLE OFF(unconditional, replaces the iOS gate)GGML_BACKEND_DL-safe registry backend init inkokoro_load_model(Android DL build)Merged FFI header lands at ABI v13 (main's additive vision streaming on top of the lineage's v12).
Verification (M4 Max, macOS)
-DGGML_METAL=ON -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_KOKORO=ON -DLLAMA_BUILD_OMNIVOICE=ON: green (log attached in the #11386 disposition).🤖 Generated with Claude Code