Skip to content

TinyML drone classifier skeleton (TFLite Micro CNN, dummy model) — closes #48 stage 1#49

Draft
TN666 wants to merge 4 commits into
mainfrom
feat/tinyml-skeleton
Draft

TinyML drone classifier skeleton (TFLite Micro CNN, dummy model) — closes #48 stage 1#49
TN666 wants to merge 4 commits into
mainfrom
feat/tinyml-skeleton

Conversation

@TN666

@TN666 TN666 commented May 3, 2026

Copy link
Copy Markdown
Member

Stage 1 of #48 — wires the TFLite Micro pipeline end-to-end with a placeholder model so we can measure flash / RAM / inference time on real hardware before training a real one. Stage 2 (real model trained on batear-io/batear-datasets) is a follow-up PR that swaps the .tflite artifact only.

Marked draft because:

  1. Hardware bring-up not done yet — needs both Heltec V3 and LILYGO boards to confirm RAM budget and inference timing.
  2. The committed model is a synthetic dummy; classification quality is meaningless until a real model ships.

Summary

Replace the FFT harmonic-ratio detector in audio_task.c with an int8 CNN running on espressif/esp-tflite-micro. The legacy harmonic path is preserved verbatim under CONFIG_BATEAR_DETECT_USE_ML=n for A/B comparison during the migration, and ML defaults to y for both detector and wired_detector roles.

Pipeline (additions to existing audio chain marked bold):

I2S 32-bit mono @ 16 kHz, 1024-sample frames     (existing)
  → audio_processor_compute_psd()                (existing — no double FFT)
  → **40-band Slaney mel filter bank + log10 + int8 quant**  (audio_features.c)
  → **[1, 32, 40, 1] int8 mel-spectrogram, 32-frame ring**   (audio_features.c)
  → **tflite::MicroInterpreter (8 registered ops)**          (ml_classifier.cpp)
  → **softmax → P(drone)**
  → same EMA + sustain hysteresis as today                   (audio_task.c)
  → DRONE_EVENT_ALARM / DRONE_EVENT_CLEAR queue              (existing)

The DroneEvent_t queue, LoRa packet format, MQTT topics, OLED, HA Discovery, and TF recording are all untouched — the change is fully contained in audio_task.c and the two new modules.

What changes

  • New modules
    • main/audio_features.{c,h} — mel filter bank + 32-frame circular int8 ring; quant params pulled from the model's input tensor at init (not hard-coded), so a re-trained model with different quant just works.
    • main/ml_classifier.{cpp,h} — C-callable wrapper around MicroInterpreter. Heap arena prefers PSRAM (LILYGO) and falls back to internal SRAM (Heltec V3). ml_classifier_init validates input shape + dtype against the firmware's expected schema and refuses to start audio task on mismatch — guards against silent shape drift when the model artifact changes.
  • Build
    • main/CMakeLists.txt — embed models/drone_v0_dummy.tflite via EMBED_FILES, pull espressif__esp-tflite-micro into PRIV_DEPS for detector + wired_detector only (gateway image untouched), hard-fail configure with a pointer to tools/gen_dummy_model.py if the model is missing.
    • main/idf_component.yml — add espressif/esp-tflite-micro (which itself pulls esp-nn).
  • Config
    • main/Kconfig.projbuild — new "TinyML detector (TFLite Micro)" submenu under detector / wired_detector roles. BATEAR_DETECT_USE_ML (master toggle), BATEAR_ML_TENSOR_ARENA_KB (100 SRAM / 200 PSRAM), BATEAR_ML_CONF_ON_PCT/_OFF_PCT (mirror legacy CONF_ON/OFF), BATEAR_TF_BORDERLINE_* (Kconfig-only stage-2 hook for active learning).
    • sdkconfig.detector — ML on, 100 KB SRAM arena.
    • sdkconfig.wired_detector — ML on, 200 KB PSRAM arena, borderline capture defined-but-disabled.
  • Audio task
    • main/audio_task.c — new ML branch behind CONFIG_BATEAR_DETECT_USE_ML. Init order: audio_processor_initml_classifier_initaudio_features_init (using model's quant params). Inference is gated by the existing RMS_MIN and by audio_features_is_warm() so silence costs zero invokes and the first ~3.2 s after boot doesn't classify a half-empty mel ring. AUDIO_PERF_LOG now reports feature-extraction and inference timings separately every 100 frames.
  • Tooling
    • tools/gen_dummy_model.py — Conv2D → DWConv → 1×1 Conv → GAP → Dense(2) Keras model, INT8 PTQ with a synthetic representative dataset, prints input/output quant params after conversion so they can be cross-checked against ml_classifier_init log output.
    • tools/requirements.txttensorflow>=2.15, numpy>=1.26.
    • models/drone_v0_dummy.tflite — committed (~4.6 KB) so the build works without running TF.

What does NOT change

  • Detector LoRa packet format and gateway RX path.
  • Wired detector MQTT / HA Discovery / REST API / TF recording behaviour.
  • Telemetry heartbeat, battery ADC, OLED.
  • The legacy harmonic detector — set CONFIG_BATEAR_DETECT_USE_ML=n and the build is byte-identical to pre-skeleton.

Test plan

Build (CI + local)

  • idf.py -B build_detector ... build with CONFIG_BATEAR_DETECT_USE_ML=y (default).
  • idf.py -B build_detector ... build with CONFIG_BATEAR_DETECT_USE_ML=n (legacy harmonic).
  • idf.py -B build_wired_detector ... build with =y.
  • idf.py -B build_wired_detector ... build with =n.
  • idf.py -B build_gateway ... build — no regression, no esp-tflite-micro in the image.
  • cppcheck workflow passes on the two new C/C++ TUs.

Hardware (Heltec V3, no PSRAM)

  • Boot logs tensor arena: 100 KB in internal SRAM and init OK: model=… arena=…/… B used input=[1,32,40,1].
  • Audio task does not suspend with ml_classifier_init failed.
  • BATEAR_AUDIO_PERF_LOG=y: per-frame inference fits well within the 100 ms hop.
  • OTA image fits in the 1 MiB OTA partition.
  • LoRa heartbeat + ALARM/CLEAR events still reach the gateway.
  • arena_used_bytes() reported at init — used to right-size the stage-2 model budget.

Hardware (LILYGO T-ETH-Lite S3, 8 MB PSRAM)

  • Boot logs tensor arena: 200 KB in PSRAM.
  • Wired detector publishes the existing MQTT status topic — no regression.
  • TF recording (PR TF card audio recording on Wired Detector (closes #46) #47) still triggers on DRONE_EVENT_ALARM and BOOT push-to-talk.
  • OTA image fits in the 1 MiB OTA partition.
  • HTTP API (/api/info, /api/status, recordings) still responds.

Schema-drift check

  • Manually corrupt models/drone_v0_dummy.tflite (e.g. change the input shape to [1, 30, 40, 1] and re-export) → boot logs model input mismatch: … and audio task suspends instead of crashing or producing garbage.

Out of scope (follow-ups)

Closes #48 (stage 1).

Made with Cursor

TN666 and others added 4 commits May 3, 2026 20:55
Stage-1 placeholder model so the firmware-side TFLite Micro pipeline can
be brought up, sized, and validated on real hardware before a real model
exists. Schema is fixed so a stage-2 trained model is a drop-in replacement.

- tools/gen_dummy_model.py: builds a tiny Conv2D + DWConv + GAP + Dense(2)
  Keras model, INT8 post-training quantised with a synthetic representative
  dataset (1, 32, 40, 1) → (1, 2). Prints input/output quant params after
  conversion so they can be cross-checked against ml_classifier_init().
- tools/requirements.txt: tensorflow + numpy pin for stage 1.
- models/drone_v0_dummy.tflite: ~4.6 KB int8 model committed so the build
  works without running the generator.
- .gitignore: skip tools/.venv and tooling caches.

Co-authored-by: Cursor <cursoragent@cursor.com>
Wires the menuconfig surface and component-manager dependency so the
detector / wired_detector roles can build with TFLite Micro.

- Kconfig.projbuild: new "TinyML detector (TFLite Micro)" submenu under
  detector / wired_detector roles. Knobs:
    BATEAR_DETECT_USE_ML            (master toggle, default y)
    BATEAR_ML_TENSOR_ARENA_KB       (100 KB SRAM / 200 KB PSRAM)
    BATEAR_ML_CONF_ON_PCT/OFF_PCT   (hysteresis, mirrors legacy CONF_ON/OFF)
    BATEAR_TF_BORDERLINE_*          (stage-2 active-learning hook —
                                     Kconfig only for now, no firmware path
                                     yet so it is a no-op at runtime).
- idf_component.yml: pull espressif/esp-tflite-micro (which itself pulls
  esp-nn for SIMD int8 kernels).
- sdkconfig.detector / sdkconfig.wired_detector: enable ML on by default
  with role-appropriate arena size (internal SRAM on Heltec V3, PSRAM
  on LILYGO). Legacy harmonic detector remains accessible via
  CONFIG_BATEAR_DETECT_USE_ML=n for A/B comparison during the migration.

Co-authored-by: Cursor <cursoragent@cursor.com>
New translation units for the TinyML skeleton. No call sites yet — the
audio_task wire-up is the next commit so each step is self-contained.

audio_features.{c,h}
- 40-band Slaney-style triangular mel filter bank over the existing
  audio_processor PSD (16 kHz, 1024-sample windows).
- 32-frame circular buffer of int8-quantised log-mel columns, output as
  a row-major [1, 32, 40, 1] grid for the classifier.
- Quantisation params are taken from the model's input tensor (via
  ml_classifier_input_quant) so log-mel and the model stay in lock-step
  even after the dummy model is replaced.
- All buffers static — no malloc on the audio path. Owns its own FFT
  scratch (~8 KB) so audio_task does not need to share its private one.

ml_classifier.{cpp,h}
- C-callable shim around tflite::MicroInterpreter so audio_task.c (a C
  TU) does not have to compile against TFLite Micro headers.
- MicroMutableOpResolver registers only the 8 ops the dummy + the
  planned stage-2 CNN need (Conv2D, DepthwiseConv2D, MaxPool2D, AvgPool2D,
  Mean, FullyConnected, Reshape, Softmax).
- Tensor arena allocated heap-side, prefers PSRAM when available else
  internal SRAM, so the same code works for Heltec V3 and LILYGO.
- Validates the model input shape against AUDIO_FEATURES_MEL_FRAMES x
  AUDIO_FEATURES_MEL_BANDS x int8 — guards against silent shape drift
  if a future model file goes out of sync with the firmware schema.
- Output supports both 2-class softmax and 1-logit sigmoid — whichever
  convention a future trained model uses, the C code just works.

Co-authored-by: Cursor <cursoragent@cursor.com>
End-to-end skeleton: I2S → mel-spectrogram → int8 CNN → drone probability →
existing EMA/sustain hysteresis → ALARM/CLEAR events. Behavioural surface
of the state machine is unchanged — only the source of detect_conf moves
from analyze_harmonics() to ml_classifier_classify().

main/CMakeLists.txt
- Compile audio_features.c and ml_classifier.cpp into both detector and
  wired_detector roles.
- Embed models/drone_v0_dummy.tflite via EMBED_FILES so the model lives
  in flash rodata and is reachable as
  _binary_drone_v0_dummy_tflite_{start,end}.
- Hard-fail the configure step with a pointer to tools/gen_dummy_model.py
  if the model file is missing — much friendlier than a downstream linker
  error.
- Pull espressif__esp-tflite-micro into PRIV_DEPS for the two roles that
  actually run inference. Gateway is unchanged.

main/audio_task.c
- New ML branch behind CONFIG_BATEAR_DETECT_USE_ML (default y):
    * ml_classifier_init() → ml_classifier_input_quant() → audio_features_init()
      at task startup; suspend on failure with a clear error log.
    * Per frame: audio_features_push_frame() does window+FFT+PSD+mel+quant
      in one call; once the 32-frame ring is warm, get_melgram() + classify()
      yields a drone probability fed into the EMA. No double-FFT.
    * RMS gate (RMS_MIN) reused from the legacy path so we don't burn
      cycles on inference during silence.
- Legacy harmonic path is preserved verbatim under #else for A/B
  comparison while the real model is being trained.
- Calibration / debug log lines branch on the back-end so the existing
  "cal:" output stays useful for both detectors.
- AUDIO_PERF_LOG now reports separate feature-extraction and inference
  timings so tensor-arena and op resolver tuning has direct feedback.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented May 3, 2026

Copy link
Copy Markdown

Firmware Size Report

Role Current Previous Delta
Detector 605824 B 492192 B +113632 B
Gateway 1083872 B 1083872 B 0 B
Wired Detector 1022752 B 906096 B +116656 B

Compared against firmware-latest.

idf.py size details

Detector: Total image size: 540175 bytes (.bin may be padded larger)
Gateway: Total image size: 1018215 bytes (.bin may be padded larger)
Wired Detector: Total image size: 957090 bytes (.bin may be padded larger)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] TinyML drone classifier skeleton (TFLite Micro CNN replacing FFT harmonic heuristic)

1 participant