TinyML drone classifier skeleton (TFLite Micro CNN, dummy model) — closes #48 stage 1#49
Draft
TN666 wants to merge 4 commits into
Draft
TinyML drone classifier skeleton (TFLite Micro CNN, dummy model) — closes #48 stage 1#49TN666 wants to merge 4 commits into
TN666 wants to merge 4 commits into
Conversation
Stage-1 placeholder model so the firmware-side TFLite Micro pipeline can be brought up, sized, and validated on real hardware before a real model exists. Schema is fixed so a stage-2 trained model is a drop-in replacement. - tools/gen_dummy_model.py: builds a tiny Conv2D + DWConv + GAP + Dense(2) Keras model, INT8 post-training quantised with a synthetic representative dataset (1, 32, 40, 1) → (1, 2). Prints input/output quant params after conversion so they can be cross-checked against ml_classifier_init(). - tools/requirements.txt: tensorflow + numpy pin for stage 1. - models/drone_v0_dummy.tflite: ~4.6 KB int8 model committed so the build works without running the generator. - .gitignore: skip tools/.venv and tooling caches. Co-authored-by: Cursor <cursoragent@cursor.com>
Wires the menuconfig surface and component-manager dependency so the
detector / wired_detector roles can build with TFLite Micro.
- Kconfig.projbuild: new "TinyML detector (TFLite Micro)" submenu under
detector / wired_detector roles. Knobs:
BATEAR_DETECT_USE_ML (master toggle, default y)
BATEAR_ML_TENSOR_ARENA_KB (100 KB SRAM / 200 KB PSRAM)
BATEAR_ML_CONF_ON_PCT/OFF_PCT (hysteresis, mirrors legacy CONF_ON/OFF)
BATEAR_TF_BORDERLINE_* (stage-2 active-learning hook —
Kconfig only for now, no firmware path
yet so it is a no-op at runtime).
- idf_component.yml: pull espressif/esp-tflite-micro (which itself pulls
esp-nn for SIMD int8 kernels).
- sdkconfig.detector / sdkconfig.wired_detector: enable ML on by default
with role-appropriate arena size (internal SRAM on Heltec V3, PSRAM
on LILYGO). Legacy harmonic detector remains accessible via
CONFIG_BATEAR_DETECT_USE_ML=n for A/B comparison during the migration.
Co-authored-by: Cursor <cursoragent@cursor.com>
New translation units for the TinyML skeleton. No call sites yet — the
audio_task wire-up is the next commit so each step is self-contained.
audio_features.{c,h}
- 40-band Slaney-style triangular mel filter bank over the existing
audio_processor PSD (16 kHz, 1024-sample windows).
- 32-frame circular buffer of int8-quantised log-mel columns, output as
a row-major [1, 32, 40, 1] grid for the classifier.
- Quantisation params are taken from the model's input tensor (via
ml_classifier_input_quant) so log-mel and the model stay in lock-step
even after the dummy model is replaced.
- All buffers static — no malloc on the audio path. Owns its own FFT
scratch (~8 KB) so audio_task does not need to share its private one.
ml_classifier.{cpp,h}
- C-callable shim around tflite::MicroInterpreter so audio_task.c (a C
TU) does not have to compile against TFLite Micro headers.
- MicroMutableOpResolver registers only the 8 ops the dummy + the
planned stage-2 CNN need (Conv2D, DepthwiseConv2D, MaxPool2D, AvgPool2D,
Mean, FullyConnected, Reshape, Softmax).
- Tensor arena allocated heap-side, prefers PSRAM when available else
internal SRAM, so the same code works for Heltec V3 and LILYGO.
- Validates the model input shape against AUDIO_FEATURES_MEL_FRAMES x
AUDIO_FEATURES_MEL_BANDS x int8 — guards against silent shape drift
if a future model file goes out of sync with the firmware schema.
- Output supports both 2-class softmax and 1-logit sigmoid — whichever
convention a future trained model uses, the C code just works.
Co-authored-by: Cursor <cursoragent@cursor.com>
End-to-end skeleton: I2S → mel-spectrogram → int8 CNN → drone probability →
existing EMA/sustain hysteresis → ALARM/CLEAR events. Behavioural surface
of the state machine is unchanged — only the source of detect_conf moves
from analyze_harmonics() to ml_classifier_classify().
main/CMakeLists.txt
- Compile audio_features.c and ml_classifier.cpp into both detector and
wired_detector roles.
- Embed models/drone_v0_dummy.tflite via EMBED_FILES so the model lives
in flash rodata and is reachable as
_binary_drone_v0_dummy_tflite_{start,end}.
- Hard-fail the configure step with a pointer to tools/gen_dummy_model.py
if the model file is missing — much friendlier than a downstream linker
error.
- Pull espressif__esp-tflite-micro into PRIV_DEPS for the two roles that
actually run inference. Gateway is unchanged.
main/audio_task.c
- New ML branch behind CONFIG_BATEAR_DETECT_USE_ML (default y):
* ml_classifier_init() → ml_classifier_input_quant() → audio_features_init()
at task startup; suspend on failure with a clear error log.
* Per frame: audio_features_push_frame() does window+FFT+PSD+mel+quant
in one call; once the 32-frame ring is warm, get_melgram() + classify()
yields a drone probability fed into the EMA. No double-FFT.
* RMS gate (RMS_MIN) reused from the legacy path so we don't burn
cycles on inference during silence.
- Legacy harmonic path is preserved verbatim under #else for A/B
comparison while the real model is being trained.
- Calibration / debug log lines branch on the back-end so the existing
"cal:" output stays useful for both detectors.
- AUDIO_PERF_LOG now reports separate feature-extraction and inference
timings so tensor-arena and op resolver tuning has direct feedback.
Co-authored-by: Cursor <cursoragent@cursor.com>
Firmware Size Report
idf.py size detailsDetector: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of #48 — wires the TFLite Micro pipeline end-to-end with a placeholder model so we can measure flash / RAM / inference time on real hardware before training a real one. Stage 2 (real model trained on
batear-io/batear-datasets) is a follow-up PR that swaps the.tfliteartifact only.Marked draft because:
Summary
Replace the FFT harmonic-ratio detector in
audio_task.cwith an int8 CNN running onespressif/esp-tflite-micro. The legacy harmonic path is preserved verbatim underCONFIG_BATEAR_DETECT_USE_ML=nfor A/B comparison during the migration, and ML defaults toyfor both detector and wired_detector roles.Pipeline (additions to existing audio chain marked bold):
The
DroneEvent_tqueue, LoRa packet format, MQTT topics, OLED, HA Discovery, and TF recording are all untouched — the change is fully contained inaudio_task.cand the two new modules.What changes
main/audio_features.{c,h}— mel filter bank + 32-frame circular int8 ring; quant params pulled from the model's input tensor at init (not hard-coded), so a re-trained model with different quant just works.main/ml_classifier.{cpp,h}— C-callable wrapper aroundMicroInterpreter. Heap arena prefers PSRAM (LILYGO) and falls back to internal SRAM (Heltec V3).ml_classifier_initvalidates input shape + dtype against the firmware's expected schema and refuses to start audio task on mismatch — guards against silent shape drift when the model artifact changes.main/CMakeLists.txt— embedmodels/drone_v0_dummy.tfliteviaEMBED_FILES, pullespressif__esp-tflite-microinto PRIV_DEPS for detector + wired_detector only (gateway image untouched), hard-fail configure with a pointer totools/gen_dummy_model.pyif the model is missing.main/idf_component.yml— addespressif/esp-tflite-micro(which itself pullsesp-nn).main/Kconfig.projbuild— new "TinyML detector (TFLite Micro)" submenu under detector / wired_detector roles.BATEAR_DETECT_USE_ML(master toggle),BATEAR_ML_TENSOR_ARENA_KB(100 SRAM / 200 PSRAM),BATEAR_ML_CONF_ON_PCT/_OFF_PCT(mirror legacyCONF_ON/OFF),BATEAR_TF_BORDERLINE_*(Kconfig-only stage-2 hook for active learning).sdkconfig.detector— ML on, 100 KB SRAM arena.sdkconfig.wired_detector— ML on, 200 KB PSRAM arena, borderline capture defined-but-disabled.main/audio_task.c— new ML branch behindCONFIG_BATEAR_DETECT_USE_ML. Init order:audio_processor_init→ml_classifier_init→audio_features_init(using model's quant params). Inference is gated by the existingRMS_MINand byaudio_features_is_warm()so silence costs zero invokes and the first ~3.2 s after boot doesn't classify a half-empty mel ring.AUDIO_PERF_LOGnow reports feature-extraction and inference timings separately every 100 frames.tools/gen_dummy_model.py— Conv2D → DWConv → 1×1 Conv → GAP → Dense(2) Keras model, INT8 PTQ with a synthetic representative dataset, prints input/output quant params after conversion so they can be cross-checked againstml_classifier_initlog output.tools/requirements.txt—tensorflow>=2.15,numpy>=1.26.models/drone_v0_dummy.tflite— committed (~4.6 KB) so the build works without running TF.What does NOT change
CONFIG_BATEAR_DETECT_USE_ML=nand the build is byte-identical to pre-skeleton.Test plan
Build (CI + local)
idf.py -B build_detector ... buildwithCONFIG_BATEAR_DETECT_USE_ML=y(default).idf.py -B build_detector ... buildwithCONFIG_BATEAR_DETECT_USE_ML=n(legacy harmonic).idf.py -B build_wired_detector ... buildwith=y.idf.py -B build_wired_detector ... buildwith=n.idf.py -B build_gateway ... build— no regression, noesp-tflite-microin the image.Hardware (Heltec V3, no PSRAM)
tensor arena: 100 KB in internal SRAMandinit OK: model=… arena=…/… B used input=[1,32,40,1].ml_classifier_init failed.BATEAR_AUDIO_PERF_LOG=y: per-frame inference fits well within the 100 ms hop.arena_used_bytes()reported at init — used to right-size the stage-2 model budget.Hardware (LILYGO T-ETH-Lite S3, 8 MB PSRAM)
tensor arena: 200 KB in PSRAM.DRONE_EVENT_ALARMand BOOT push-to-talk./api/info,/api/status, recordings) still responds.Schema-drift check
models/drone_v0_dummy.tflite(e.g. change the input shape to[1, 30, 40, 1]and re-export) → boot logsmodel input mismatch: …and audio task suspends instead of crashing or producing garbage.Out of scope (follow-ups)
batear-io/batear-datasets(alarm WAVs from PR TF card audio recording on Wired Detector (closes #46) #47 + ambient negatives), drop in as a newmodels/drone_*.tflite, no firmware changes if shapes match.audio_task.ctotf_recorder.cfor borderlineP(drone) ∈ [BORDERLINE_LO_PCT, BORDERLINE_HI_PCT]samples, writing_borderline.wav. Kconfig is already defined; only the firmware path remains.ml_invoke_us_avgdiagnostic sensor — exposed via HA for ongoing perf monitoring on wired detector.Closes #48 (stage 1).
Made with Cursor