|
| 1 | +# Session-op backend seam — design (NOT implemented) |
| 2 | + |
| 3 | +The per-op backend seam (`backend-registry.h` + `<mod>-backend.h` + |
| 4 | +`<mod>-backend-selector.cpp` + a chokepoint at the top of the FFI fn) is now in |
| 5 | +place for the **one-shot** ops: |
| 6 | + |
| 7 | +| modality | FFI fn | header / selector | env key | artifact dir | |
| 8 | +|----------|---------------------------------|------------------------------|-----------------------|--------------------| |
| 9 | +| embed | `eliza_inference_embed` | `embed-backend.*` | `ELIZA_EMBED_BACKEND` | `<bundle>/embedding/` | |
| 10 | +| vision | `eliza_inference_describe_image`| `vision-backend.*` | `ELIZA_VISION_BACKEND`| `<bundle>/vision/` | |
| 11 | +| asr | `eliza_inference_asr_transcribe`| `asr-backend.*` | `ELIZA_ASR_BACKEND` | `<bundle>/asr/` | |
| 12 | +| tts | `eliza_inference_tts_synthesize`| `tts-backend.*` | `ELIZA_TTS_BACKEND` | `<bundle>/tts/` | |
| 13 | +| eot | `eliza_inference_llm_eot_score` | `eot-backend.*` | `ELIZA_EOT_BACKEND` | `<bundle>/eot/` | |
| 14 | + |
| 15 | +A one-shot op is stateless across calls: select → (delegate | fall through to |
| 16 | +ggml) on every call. There is nothing to keep alive between calls, so the seam |
| 17 | +is a single chokepoint at the top of the fn. |
| 18 | + |
| 19 | +The **session** ops are different: `vad`, `wakeword`, `speaker`, `diariz` each |
| 20 | +`_open` a native handle (`EliVad *`, `EliWakeword *`, `EliSpeaker *`, |
| 21 | +`EliDiariz *`) that persists across many `_segment`/`_detect`/`_embed` calls and |
| 22 | +is torn down with `_close`/`_reset`. The seam has to follow that lifecycle, not |
| 23 | +re-select per call. This file records HOW to extend the seam to them. **None of |
| 24 | +the below is implemented yet.** |
| 25 | + |
| 26 | +## The shape of a session op (today, in-tree only) |
| 27 | + |
| 28 | +Each session modality exposes, e.g. for VAD: |
| 29 | + |
| 30 | +```c |
| 31 | +EliVad * eliza_inference_vad_open(EliInferenceContext * ctx, /* params */, char ** out_error); |
| 32 | +int eliza_inference_vad_segment(EliVad * vad, const float * pcm, size_t n, /* out */, char ** out_error); |
| 33 | +int eliza_inference_vad_reset(EliVad * vad, char ** out_error); |
| 34 | +void eliza_inference_vad_close(EliVad * vad); |
| 35 | +``` |
| 36 | +
|
| 37 | +`EliVad` (and the wakeword/speaker/diariz equivalents) is the in-tree handle |
| 38 | +struct defined in `eliza-inference-ffi.cpp`. Its in-tree fields stay exactly as |
| 39 | +they are; the seam is **additive** — one extra pointer. |
| 40 | +
|
| 41 | +## Extending the seam to a session op |
| 42 | +
|
| 43 | +For each session modality `<mod>` (vad | wakeword | speaker | diariz): |
| 44 | +
|
| 45 | +### 1. A session factory interface — `<mod>-backend.h` |
| 46 | +
|
| 47 | +Mirror the one-shot factory's four common probes, but the forward methods mirror |
| 48 | +the **session** ABI 1:1 instead of a single one-shot fn. The factory does NOT |
| 49 | +own the handle struct; it produces and operates on an opaque backend-session: |
| 50 | +
|
| 51 | +```cpp |
| 52 | +struct VadBackendFactory { |
| 53 | + virtual ~VadBackendFactory() = default; |
| 54 | + virtual const char * name() const = 0; |
| 55 | + virtual bool available() const = 0; |
| 56 | + virtual bool can_serve(const char * bundle_dir) const = 0; // probes <bundle>/vad/ |
| 57 | + virtual int preference_rank() const { return 0; } |
| 58 | +
|
| 59 | + // Lifecycle, mirroring the FFI session ABI 1:1. The factory returns an |
| 60 | + // opaque backend-session pointer it owns; the FFI stashes it on the Eli* |
| 61 | + // handle. A NULL return + *out_error is a hard open failure. |
| 62 | + virtual void * open(EliInferenceContext * ctx, /* same params as eliza_inference_vad_open */, |
| 63 | + char ** out_error) = 0; |
| 64 | + virtual int segment(void * session, const float * pcm, size_t n, /* out */, char ** out_error) = 0; |
| 65 | + virtual int reset(void * session, char ** out_error) = 0; |
| 66 | + virtual void close(void * session) = 0; |
| 67 | +}; |
| 68 | +``` |
| 69 | + |
| 70 | +Plus the same free-functions as the one-shot seam: |
| 71 | +`vad_backend_register`, `vad_backend_register_builtins` (EMPTY for now — no |
| 72 | +LiteRT session backend exists), `vad_backend_select(bundle_dir, out_error)`, |
| 73 | +backed by a `eliza_backend::Registry<VadBackendFactory>` in |
| 74 | +`<mod>-backend-selector.cpp` with env keys `ELIZA_VAD_BACKEND` → `ELIZA_BACKEND` |
| 75 | +and modality `"vad"`. Artifact probe dir `<bundle>/vad/` (resp. `wakeword/`, |
| 76 | +`speaker/`, `diariz/`). |
| 77 | + |
| 78 | +### 2. A backend-session pointer on the Eli* handle |
| 79 | + |
| 80 | +The selection happens ONCE, at `_open`, not per call. Add one field to the |
| 81 | +in-tree handle struct: |
| 82 | + |
| 83 | +```cpp |
| 84 | +struct EliVad { |
| 85 | + /* ... existing in-tree fields, unchanged ... */ |
| 86 | + |
| 87 | + /* Backend seam (additive). When non-null, this handle is served by an |
| 88 | + * accelerator backend and every op delegates to it; the in-tree fields |
| 89 | + * above are then unused. When null, the in-tree ggml path owns the handle. */ |
| 90 | + VadBackendFactory * be = nullptr; // the factory that opened be_session |
| 91 | + void * be_session = nullptr; // factory-owned backend session |
| 92 | +}; |
| 93 | +``` |
| 94 | +
|
| 95 | +### 3. Select at `_open` |
| 96 | +
|
| 97 | +In `eliza_inference_vad_open`, after the existing arg validation and before the |
| 98 | +in-tree handle is built: |
| 99 | +
|
| 100 | +```cpp |
| 101 | +char * be_error = nullptr; |
| 102 | +VadBackendFactory * be = vad_backend_select(llm_backend_context_bundle_dir(ctx), &be_error); |
| 103 | +if (be_error) { eliza_set_error(out_error, std::string(be_error)); std::free(be_error); |
| 104 | + return /* NULL handle */; } |
| 105 | +if (be) { |
| 106 | + void * sess = be->open(ctx, /* params */, out_error); |
| 107 | + if (!sess) return /* NULL handle — open failed, out_error already set */; |
| 108 | + EliVad * h = new EliVad(); |
| 109 | + h->be = be; |
| 110 | + h->be_session = sess; |
| 111 | + return h; |
| 112 | +} |
| 113 | +/* else: fall through and build the in-tree handle exactly as today. */ |
| 114 | +``` |
| 115 | + |
| 116 | +### 4. A guard at the TOP of each `_segment` / `_reset` / `_close` |
| 117 | + |
| 118 | +Each per-call op checks the backend pointer and delegates before touching any |
| 119 | +in-tree state: |
| 120 | + |
| 121 | +```cpp |
| 122 | +int eliza_inference_vad_segment(EliVad * vad, const float * pcm, size_t n, /* out */, char ** out_error) { |
| 123 | + if (!vad) { /* invalid-arg as today */ } |
| 124 | + if (vad->be) { // <-- guard |
| 125 | + return vad->be->segment(vad->be_session, pcm, n, /* out */, out_error); |
| 126 | + } |
| 127 | + /* ... existing in-tree ggml segment body, unchanged ... */ |
| 128 | +} |
| 129 | + |
| 130 | +void eliza_inference_vad_close(EliVad * vad) { |
| 131 | + if (!vad) return; |
| 132 | + if (vad->be) { vad->be->close(vad->be_session); delete vad; return; } // <-- guard |
| 133 | + /* ... existing in-tree teardown, then delete vad ... */ |
| 134 | +} |
| 135 | +``` |
| 136 | +
|
| 137 | +`_reset` follows the same guard pattern. |
| 138 | +
|
| 139 | +## Why this shape (vs. re-selecting per call) |
| 140 | +
|
| 141 | +- **Selection is per-session, not per-call.** A session's backend is fixed at |
| 142 | + `_open`; you cannot have `_segment` cross from the ggml path to LiteRT mid |
| 143 | + session because the KV/feature state lives in the (in-tree OR backend) |
| 144 | + session, not on the FFI boundary. The one pointer captures that binding. |
| 145 | +- **Hard-fail localizes to `_open`.** A bundle-invalid override surfaces once, |
| 146 | + where the caller is already prepared to handle a NULL handle, instead of on |
| 147 | + every `_segment`. |
| 148 | +- **Additive + inert.** With no session backend registered (the case today), |
| 149 | + `_open`'s `select()` returns nullptr, `be`/`be_session` stay null, and every |
| 150 | + guard is a no-op — the in-tree path is byte-for-byte unchanged. Same inert-by |
| 151 | + -default contract as the one-shot seam. |
| 152 | +
|
| 153 | +## Status |
| 154 | +
|
| 155 | +- One-shot seam: embed (with a LiteRT builtin), vision/asr/tts/eot (inert, |
| 156 | + no builtin) — **done**. |
| 157 | +- Session seam (vad/wakeword/speaker/diariz): **not implemented.** No |
| 158 | + `<mod>-backend.{h,cpp}`, no handle field, no `_open` select, no per-call |
| 159 | + guards exist yet. This file is the spec for when a session backend lands. |
0 commit comments