Fix media pipelines, recording lifecycle, signaling, and service installation

Preserve cross-platform capture and encoder fallbacks, validate configuration and signaling inputs, clean up failed recording setup, and improve operational guides. Fix portable HLS serving and room NDI codec dispatch. Validated with 149 application tests, 218 external regressions, and focused Pi 3/Pi 5 checks.

Author: steveseguin

AGENTS.md

@@ -24,3 +24,14 @@ When changing installers, device discovery, camera handling, media pipelines, or
 - Add focused regression coverage for detection, selection, fallback, and version-specific behavior whenever practical.
 
 Treat broad support across these variations as a core product requirement rather than an optional enhancement.
+
+## Hardware and regression testing
+
+- Physical Raspberry Pi boards are available on the network. Use the sibling
+  `../raspberry_ninja_qa` repository's local inventory and SSH configuration to
+  discover and test reachable boards; do not assume hardware is unavailable.
+- Keep new hardware harnesses, integration regressions, scaffolding, and generated
+  reports in that QA repository. Keep the application repository lean; retain
+  existing focused tests and avoid duplicating the external harness here.
+- Read the QA repository's instructions before changing it. Keep credentials and
+  local device addresses out of committed documentation and reports.

QUICK_START.md

@@ -48,6 +48,9 @@ selected service. Advanced service options are documented by
 
 ## Optional one-time test
 
+For reusable manual commands, see
+[JSON configuration and command-line overrides](docs/operations-guide.md#save-settings-in-a-json-configuration).
+
 To publish a small test pattern without changing the saved setup:
 
 ```bash

README.md

@@ -811,6 +811,10 @@ python3 publish.py --view STREAMIDHERE --v4l2sink 0 \
 
 Notes:
 - `--v4l2sink` accepts a numeric index (`0`) or a full path (`/dev/video2`).
+- Select a writable video output such as a configured `v4l2loopback` device.
+  Raspberry Pi ISP and codec nodes are internal processing devices, not virtual
+  cameras, and are excluded from output discovery. If no suitable output exists,
+  configure a loopback device first and check it with `v4l2-ctl -d /dev/video2 -D`.
 - If the specified device is not writable, the first writable `/dev/video*` is used.
 - When no remote video is available, a blue frame is output to keep the device alive.
 - The V4L2 sink path drops upstream allocation queries before `v4l2sink` to avoid buffer-pool issues with some `v4l2loopback` versions. The default V4L2 sink I/O mode is GStreamer's auto mode (`--v4l2sink-io-mode 0`); use `--v4l2sink-io-mode 1` to force read/write mode.

config_loader.py

@@ -2,12 +2,18 @@
 
 from argparse import ArgumentParser, Namespace
 import json
+import math
 from pathlib import Path
 from typing import Any, Dict, Iterable, Optional, Set
 
 
 IGNORED_CONFIG_KEYS = {"platform", "auto_start", "custom_video_pipeline", "video_device", "video_format"}
 CONFIG_ARG_ALIASES = {"stream_id": "streamid"}
+AUDIO_SOURCE_OVERRIDE_ATTRS = {"alsa", "pulse", "audio_pipeline", "noaudio"}
+VIDEO_CODEC_OVERRIDE_ATTRS = {
+    "h264", "x264", "openh264", "omx", "vp8", "vp9",
+    "h265", "hevc", "x265", "av1", "aom", "rav1e", "qsv",
+}
 VIDEO_SOURCE_OVERRIDE_ATTRS = (
     "test",
     "hdmi",
@@ -31,7 +37,7 @@
 def load_config_file(path: str) -> Dict[str, Any]:
     """Load one JSON configuration file or raise an actionable exception."""
     config_path = Path(path).expanduser()
-    with config_path.open("r", encoding="utf-8") as config_file:
+    with config_path.open("r", encoding="utf-8-sig") as config_file:
         config = json.load(config_file)
     if not isinstance(config, dict):
         raise ValueError("configuration root must be a JSON object")
@@ -47,10 +53,18 @@ def _explicit_cli_destinations(parser: ArgumentParser, cli_argv: Iterable[str])
     }
     destinations: Set[str] = set()
     for token in cli_argv:
+        if token == "--":
+            break
         if not token.startswith("-"):
             continue
         option = token.split("=", 1)[0]
         destination = option_destinations.get(option)
+        if destination is None and parser.allow_abbrev and option.startswith("--"):
+            # argparse accepts unique long-option prefixes. Preserve explicit
+            # values even when they happen to equal the parser's default.
+            matches = [name for name in option_destinations if name.startswith(option)]
+            if len(matches) == 1:
+                destination = option_destinations[matches[0]]
         if destination:
             destinations.add(destination)
     return destinations
@@ -92,16 +106,49 @@ def _apply_video_source_override(
     if _video_source_has_cli_override(args, parser, explicit_cli_args):
         return
 
+    if value not in ("test", "libcamera", "v4l2", "custom"):
+        parser.error("configuration key 'video_source' must be one of: test, libcamera, v4l2, custom")
+
     if value == "test" and _arg_is_default(args, parser, "test", explicit_cli_args):
         args.test = True
     elif value == "libcamera" and _arg_is_default(args, parser, "libcamera", explicit_cli_args):
         args.libcamera = True
     elif value == "v4l2" and _arg_is_default(args, parser, "v4l2", explicit_cli_args):
-        args.v4l2 = config.get("video_device", "/dev/video0")
+        device = config.get("video_device", "/dev/video0")
+        if not isinstance(device, str) or not device.strip():
+            parser.error("configuration key 'video_device' must be a non-empty JSON string")
+        args.v4l2 = device
     elif value == "custom" and _arg_is_default(args, parser, "video_pipeline", explicit_cli_args):
         custom_pipeline = config.get("custom_video_pipeline")
-        if custom_pipeline:
-            args.video_pipeline = custom_pipeline
+        if not isinstance(custom_pipeline, str) or not custom_pipeline.strip():
+            parser.error("configuration key 'custom_video_pipeline' must be a non-empty JSON string when video_source is custom")
+        args.video_pipeline = custom_pipeline
+
+
+def _validate_config_value(parser: ArgumentParser, action, key: str, value: Any) -> None:
+    """Check typed settings before bypassing argparse via setattr."""
+    if value is None and action.default is None:
+        return
+    if action.nargs == 0 and isinstance(action.const, bool):
+        if not isinstance(value, bool):
+            parser.error(f"configuration key '{key}' must be a JSON boolean (true or false)")
+    elif action.type is int:
+        if type(value) is not int:
+            parser.error(f"configuration key '{key}' must be a JSON integer")
+    elif action.type is str:
+        if not isinstance(value, str):
+            parser.error(f"configuration key '{key}' must be a JSON string")
+    elif action.type is float:
+        if type(value) not in (int, float):
+            parser.error(f"configuration key '{key}' must be a finite JSON number")
+        try:
+            finite = math.isfinite(value)
+        except OverflowError:
+            finite = False
+        if not finite:
+            parser.error(f"configuration key '{key}' must be a finite JSON number")
+    if action.choices is not None and value not in action.choices:
+        parser.error(f"configuration key '{key}' must be one of: " + ", ".join(map(str, action.choices)))
 
 
 def apply_config_overrides(
@@ -111,22 +158,45 @@ def apply_config_overrides(
     cli_argv: Optional[Iterable[str]] = None,
 ) -> Namespace:
     explicit_cli_args = _explicit_cli_destinations(parser, cli_argv or ())
+    # Snapshot before applying config values, which must not be mistaken for
+    # command-line source selections on later iterations.
+    cli_video_source = _video_source_has_cli_override(args, parser, explicit_cli_args)
+    cli_video_codec = bool(explicit_cli_args & VIDEO_CODEC_OVERRIDE_ATTRS)
+    cli_audio_source = bool(explicit_cli_args & AUDIO_SOURCE_OVERRIDE_ATTRS)
+    actions = {action.dest: action for action in parser._actions}
 
     for key, value in config.items():
         if key in IGNORED_CONFIG_KEYS:
             continue
 
         if key == "audio_enabled":
-            if value is False and _arg_is_default(args, parser, "noaudio", explicit_cli_args):
-                args.noaudio = True
+            if cli_audio_source or "noaudio" in config:
+                continue
+            if _arg_is_default(args, parser, "noaudio", explicit_cli_args):
+                if not isinstance(value, bool):
+                    parser.error("configuration key 'audio_enabled' must be a JSON boolean (true or false)")
+                if value is False:
+                    args.noaudio = True
             continue
 
         if key == "video_source":
             _apply_video_source_override(args, parser, value, config, explicit_cli_args)
             continue
 
         target_key = CONFIG_ARG_ALIASES.get(key, key)
+        # Prefer current names over legacy aliases independently of JSON order.
+        if target_key != key and target_key in config:
+            continue
+        if cli_audio_source and target_key in AUDIO_SOURCE_OVERRIDE_ATTRS:
+            continue
+        if cli_video_codec and target_key in VIDEO_CODEC_OVERRIDE_ATTRS:
+            continue
+        if cli_video_source and target_key in VIDEO_SOURCE_OVERRIDE_ATTRS:
+            continue
         if _arg_is_default(args, parser, target_key, explicit_cli_args):
+            action = actions.get(target_key)
+            if action is not None:
+                _validate_config_value(parser, action, key, value)
             setattr(args, target_key, value)
 
     return args

docs/operations-guide.md

@@ -58,6 +58,15 @@ python3 -u publish.py \
 
 VP9 is software-heavy on small boards. Validate it at a low resolution before increasing load.
 
+## Multiple viewers and stalled connections
+
+Use `--multiviewer` to share one encoded stream with multiple viewers. Each
+viewer has bounded audio and video queues. If one viewer stops accepting media,
+its queues drop old packets when full so the other viewers can keep receiving.
+The affected viewer may have audio gaps or need the next video keyframe when it
+recovers. This does not increase the publisher's available upload bandwidth;
+choose a bitrate that leaves room for all viewers.
+
 ## Publish a USB camera or HDMI capture device
 
 List devices and the modes of the intended capture node:
@@ -131,6 +140,80 @@ The default preserves aspect ratio on whatever mode the display advertises. Add
 
 The receiver remains available while the sender is absent. Automatic retry defaults to a short sequence followed by a longer interval. Use `--no-auto-retry` only for a supervised diagnostic run.
 
+After a detected disconnect, `--viewer-retry-initial` waits 15 seconds by default
+before the first play request. The next request waits `--viewer-retry-short`
+(45 seconds), and later requests use `--viewer-retry-long` (180 seconds).
+Set `--viewer-retry-initial 0` for an immediate first retry. These intervals govern
+viewer play requests, separately from reconnecting to the signaling server.
+If a request cannot be scheduled because the signaling loop is unavailable, it
+does not advance the retry count; another attempt is scheduled after the long
+delay. Successful peer creation resets the retry sequence.
+
+## Save settings in a JSON configuration
+
+Create `sender.json` with JSON booleans and numbers (without quotes):
+
+```json
+{
+  "streamid": "my-camera",
+  "password": "replace-with-your-shared-password",
+  "test": true,
+  "h264": true,
+  "noaudio": true,
+  "width": 640,
+  "height": 360,
+  "framerate": 15,
+  "bitrate": 500
+}
+```
+
+Run `python3 publish.py --config sender.json`. Use the same stream ID and
+password in the viewer. Protect files containing passwords with
+`chmod 600 sender.json`; do not post them in bug reports.
+
+Keys normally use argument destination names: `streamid`, `noaudio`, and
+`video_pipeline`, for example. Installer-style `stream_id` is also accepted.
+If both names are present, `streamid` takes precedence over `stream_id` regardless
+of JSON key order. Likewise, `noaudio` takes precedence over legacy
+`audio_enabled`. Explicit command-line options still take precedence over the file.
+Legacy `video_source` accepts `test`, `libcamera`, `v4l2`, or `custom`.
+`custom` requires a non-empty `custom_video_pipeline`; `v4l2` uses `/dev/video0`
+when `video_device` is omitted, but rejects an explicitly empty or null device.
+Invalid source selections stop startup instead of silently falling back to a camera.
+Save as UTF-8; files with a UTF-8 byte-order mark are supported. A missing file,
+invalid JSON, or a root value other than an object stops startup with an error.
+Boolean flags require `true` or `false`; integer settings such as `bitrate`
+require integers; floating-point settings require finite numbers. Numeric strings
+such as `"500"` and boolean strings such as `"false"` are rejected rather than
+interpreted as flags or passed into media setup. Options with a fixed set of
+choices, such as `ice_transport_policy`, use the same choices as the CLI.
+Text settings, including stream IDs, passwords, device paths, and custom
+pipelines, require JSON strings. To disable the password, use `"password": "false"`
+(a string); `"noaudio": false` is a boolean flag. Optional text settings whose
+default is unset also accept `null`.
+
+Explicit command-line values override saved settings, even if the value equals
+the built-in default. For example, `--config sender.json --bitrate 2500` uses
+2500 kbps. Unique long-option abbreviations follow the same rule, but use full
+option names in scripts so future options cannot make an abbreviation ambiguous.
+
+An explicit codec or encoder flag also replaces saved codec-selection flags.
+For example, `--config sender.json --x264` selects H.264 even if the file enables
+VP8 or AV1. Other saved settings, including bitrate and platform hints, still
+apply. Without an explicit codec flag, the saved codec selection is used.
+
+Likewise, `--alsa`, `--pulse`, `--audio-pipeline`, or `--noaudio` replaces saved
+audio-source and audio-enable settings. For example, an explicit `--alsa DEVICE`
+enables that source even if the file contains `"noaudio": true`. Saved audio
+bitrate and other unrelated options still apply.
+
+Choosing a video source on the command line also suppresses saved video-source
+flags. For example, `--config sender.json --v4l2 /dev/video2` replaces the saved
+test source with that camera. Conversely, `--test` replaces a saved camera source.
+Other settings, including resolution, codec, bitrate, and password, still apply;
+ensure they are suitable for the replacement source. This changes only the current
+invocation, not the JSON file or an already-running service.
+
 ## Run unattended with systemd
 
 First prove the exact command interactively. Then use `tools/install_unattended.py` to create a validated receiver or sender unit whose user and working directory match the installed clone. Complete examples are in the [Pi Zero 2 W guide](pi-zero-2-w-unattended-webrtc.md#8-make-the-receiver-start-on-boot).
@@ -146,6 +229,60 @@ journalctl -u raspberry-ninja-viewer.service -f
 
 The helper uses `Restart=always`, a small `RestartSec`, unbuffered Python output, and `network-online.target`. It stores credentials in a restricted JSON config instead of the unit command. Running the installer again validates the replacement unit and restarts the existing service so new settings take effect.
 
+`--service-name` accepts up to 247 letters, digits, underscores, dots, hyphens,
+or `@` characters before the generated `.service` suffix. It must not start with
+`-` or `@`. Use a concrete instance such as `camera@front`, rather than `camera@`.
+
+For senders, omit `--audio-device` to disable audio; an empty value is invalid.
+`--camera` requires a non-empty device path. `--allow-missing-device` permits an
+unplugged device but does not allow an empty path or a directory.
+
+Configuration writes use a private temporary file in the destination directory,
+then replace the destination after setting its permissions and ownership. The shared
+config directory is root-owned with mode `0711` (traversal without listing), and
+each config uses mode `0640` with its service user's group. Installing another
+service under a different user therefore preserves access to existing configs. If that
+write fails, the temporary file is removed and the previous destination remains
+intact. This protects the file update; it does not guarantee that newly selected
+camera or network settings will work. Check the service status and journal after
+each reconfiguration. `--dry-run` prints the proposed configuration, including its
+password, so keep that output private.
+
+If writing or verifying the generated files fails before systemd is reloaded,
+the installer restores replaced files atomically, including their previous
+permissions and ownership, and restores the config directory's metadata.
+An incomplete rollback is reported explicitly. This rollback does not cover
+failures during the later service reload, enable, or restart steps.
+
+Relative `--python` and `--camera` paths become absolute from the installer's working
+directory without resolving virtual-environment or stable device symlinks. The generated service treats paths
+literally, including spaces, percent signs, and dollar signs. Paths containing line breaks or NUL
+bytes are rejected. Use `--dry-run` to inspect paths before installation.
+
+## Record while publishing to RTMP
+
+For RTMP publishing, `--save` also writes a local timestamped `.mkv` recording.
+The RTMP output stays connected to its muxer even if `--multiviewer` is present;
+that flag's dynamic viewer branches apply to WebRTC publishing. Verify recording
+output and available disk space before leaving a sender unattended.
+
+For a V4L2 camera that advertises H.264, `--v4l2 /dev/videoN --format H264`
+can send the camera's encoded video directly to RTMP without decoding and
+re-encoding. RTMP receives parsed H.264; WebRTC uses RTP packetization.
+Confirm the camera's advertised modes with `v4l2-ctl --list-formats-ext`.
+
+RTMP audio is converted and resampled to 48 kHz mono before AAC encoding, so a
+44.1 kHz source can be used. A custom `--audio-pipeline` supplies its own source
+and does not require a detected microphone; `--noaudio` still disables audio.
+
+Automatic audio selection uses devices that advertise an ALSA card, preferring
+a valid default device. If none is found, audio is disabled with a diagnostic.
+Use `--alsa DEVICE` or `--pulse DEVICE` when the desired source is not represented
+by an ALSA card in device discovery; these explicit selections bypass discovery.
+Pass a device name containing spaces as one shell-quoted argument. The app
+preserves the name when constructing the ALSA or PulseAudio source, including
+literal quotes and backslashes; do not add GStreamer property syntax yourself.
+
 ## Conservative performance profiles
 
 These are starting points, not guaranteed limits:
@@ -181,3 +318,19 @@ During a soak, confirm:
 
 See [Troubleshooting](troubleshooting.md) for diagnostic commands and [Recording](recording-guide.md) for file validation.
 For a repeatable deployment gate, use the [unattended validation checklist](unattended-validation-checklist.md).
+
+## Serving existing HLS files
+
+To serve an existing playlist and its segments locally:
+
+```bash
+python3 tools/serve_hls.py --directory /path/to/hls --bind 127.0.0.1 --port 8089
+```
+
+Open the playlist at `http://127.0.0.1:8089/PLAYLIST.m3u8` in an HLS-compatible
+player. The helper serves files; it does not generate the stream. It supports
+concurrent requests and CORS preflight. Playlist responses disable caching so live
+updates remain visible even when the playlist changes within one second.
+Without options it serves the repository
+root on port 8089 on all interfaces. Use `--directory` to select the files to expose
+and `--bind` to choose the listening address. Stop with Ctrl+C.