feat (future): WebRTC peer-to-peer transport with signaling-only directory

[geldoronie]

Status: future / wishlist — no milestone, no timeline. Records the architectural direction so it isn't lost. Will be considered after #49 (the stream directory) has settled in real use and we have signal that the tunnel-based exposure isn't enough.

The stream directory (#49) solves "how do users find each other" but currently relies on each host having a way to be reachable from the public internet — either port-forwarding, an integrated Cloudflare Tunnel, or a user-provided custom tunnel URL. This works but couples RetroCapture to third-party infrastructure (Cloudflare specifically) for the default path, and means home users with strict CGNAT and no cloudflared compatibility have nowhere to go.

This issue tracks a longer-term alternative: replace MPEG-TS-over-HTTP with WebRTC, with the existing directory service repurposed as a signaling-only channel. The directory never sees stream bytes (already true after #49), but additionally never sees any tunnel configuration — peers negotiate a direct connection through ICE / STUN / TURN, and the stream flows peer-to-peer.

Background — what WebRTC does that we don't

WebRTC is the umbrella name for a stack of protocols originally built for browser-to-browser real-time communication. The pieces that matter here:

• ICE (Interactive Connectivity Establishment) — RFC 8445. Tries every reasonable path between two peers (direct IP, NAT hole-punching via STUN, relayed via TURN) and picks the first one that works. This is the part that obsoletes our tunnel question. ICE handles roughly 95 % of real-world NAT scenarios automatically.
• STUN servers — free public infrastructure (Google, Cloudflare, others run them). Peers use them to discover their own public IP / port mapping. We don't pay for or operate STUN.
• TURN servers — fallback relay for the ~5 % of cases where ICE can't find a direct path (strict symmetric NAT, mostly mobile / corporate). TURN relays do consume bandwidth, so someone has to pay. We can host one, use a free-tier provider (Twilio, Metered.ca, Xirsys all offer limited free TURN), or push that cost onto the user via their own TURN config.
• SDP (Session Description Protocol) — RFC 8866. Text format that describes codecs, ports, etc. Peers exchange SDP offers / answers through the signaling channel (our directory).
• DTLS-SRTP — wire encryption built into WebRTC. Streams are encrypted end-to-end by default; the directory can't snoop even if it wanted to.
• Congestion control (GCC / SCReAM) — WebRTC encoders react to network conditions and adjust bitrate / framerate dynamically. This is a meaningful change from our current "fixed bitrate, drop on congestion" model.
• Codec subset — H.264 is universally supported. H.265 / HEVC support exists but is patchy across implementations. VP8 / VP9 / AV1 supported in browsers but our pipeline doesn't currently emit them. Important constraint when designing this: WebRTC has opinions about which H.264 / H.265 profiles and slice configurations it accepts.

High-level architecture sketch

                                  ┌──────────────────────────┐
                                  │   Directory service      │
                                  │   (signaling channel)    │
                                  └──────┬───────────────────┘
                                         │
            SDP offers / ICE candidates over WebSocket
                                         │
                          ┌──────────────┴──────────────┐
                          ▼                             ▼
                     RetroCapture host          RetroCapture client
                          │                             │
                          │  ICE handshake              │
                          │ (STUN-discovered            │
                          │  candidates)                │
                          ▼                             ▼
                          └─────────── DTLS-SRTP ───────┘
                                  (peer-to-peer)

                          Streams flow directly between peers.
                          The directory NEVER sees stream bytes.
                          No cloudflared, no FRP, no Tailscale.

Protocol shape

• Directory gains a WebSocket endpoint, wss://directory.../signaling/<streamId>, that proxies SDP offer / answer pairs and trickled ICE candidates between exactly two parties. The signaling channel is small (handful of KB per connection) and short-lived (closes once the WebRTC peer connection is up). Bandwidth cost remains negligible.
• New endpoint mode in the directory schema: endpointMode: "webrtc". The endpoint field doesn't carry a URL anymore — instead it carries the WebSocket signaling path and the public ICE / TURN configuration.
• The existing direct / tunnel-cloudflare / custom modes from feat: public stream directory — opt-in discovery for remote RetroCapture streams #49 remain valid. WebRTC is a fourth option, not a replacement. A given host picks one mode at publish time.

Per-frame data path

• Host's encoded H.264 / H.265 frames feed an RTCPeerConnection track instead of being muxed into MPEG-TS over HTTP.
• WebRTC handles packetisation (RTP), encryption (DTLS-SRTP), congestion feedback, and forward error correction.
• Client receives the RTP stream, decodes, feeds the existing frame pipeline (interpolation, shader pass, display) just as it does today after MPEG-TS demux.
• Audio rides the same peer connection on a separate track. WebRTC has explicit audio / video track concepts; this matches our existing two-stream model.

Implementation considerations

Library choice

• libdatachannel (C++, MIT, ~10k LoC). Lightweight, focused on WebRTC datachannels and media transport. Doesn't bundle a stack of unrelated browser primitives. Best fit for embedding.
• libwebrtc (Google's reference implementation, BSD, but ~500 MB of source and the build system is famously hostile). Powerful but overkill — most of it is browser-specific.
• Pion (Go) would be ideal for the signaling server but we'd still need a C++ client lib for RetroCapture itself.

Recommended: libdatachannel for the C++ side, Pion (or a Cloudflare Worker speaking the WebRTC signaling protocol) for the directory's signaling endpoint.

Codec compatibility

Our current hardware encoder dispatch (NVENC / VAAPI / QSV / AMF) emits H.264 and H.265 with whatever profile / level our config selects. WebRTC requires specific configurations:

• H.264: Constrained Baseline or Constrained High profiles. Our current Main profile output may need to be re-configured. SPS / PPS must be inline (we already do this for /raw mid-join, feat: shader-preserving distributed playback — "Connect to Remote Stream" client mode #47).
• H.265 / HEVC: WebRTC HEVC is supported by Safari (Apple) and partially by Chrome behind flags. Cross-browser interop is poor. For a desktop-only target (RetroCapture client) we can pick the implementation that supports it, but it limits future browser-client work.
• Latency: WebRTC's congestion controller will dial back framerate or bitrate under network pressure. Our current "always 60 fps, 5 Mbps" model becomes "up to 60 fps, up to 5 Mbps, adaptive". User-visible behaviour change.

Mid-stream join

WebRTC peer connections are 1:1. Multiple viewers = multiple peer connections from the host. This means:

• Host CPU cost scales linearly with viewer count (each connection runs its own DTLS / RTP state). For a few viewers this is fine; for dozens, the host machine starts to feel it.
• For broadcast-scale (hundreds of viewers) you'd want an SFU (Selective Forwarding Unit) — a middlebox that the host streams to once, and that forwards to N viewers. SFUs are real servers with real bandwidth costs. Out of scope for v1 of this idea; mention only.

Backwards compatibility

The existing /raw + /meta HTTP endpoints (from #47) keep working. A WebRTC-enabled host can simultaneously serve:

• WebRTC peers via the directory's signaling channel.
• Direct HTTP clients (e.g. someone given the URL out of band) via the existing endpoints.

Old RetroCapture builds (pre-WebRTC) just don't see the webrtc-mode entries in the directory — they show up as "unsupported" in the browse list with a tooltip.

TURN funding

The honest cost analysis:

• ICE direct-path success rate in the wild: ~85–90 % for residential broadband, ~50–70 % for mobile / strict CGNAT.
• TURN relay carries the failing percentage. Per-user bandwidth on TURN = full stream bitrate (3–5 Mbps each direction). At 100 concurrent TURN-relayed streams that's ~500 Mbps continuous.
• Free TURN options (Twilio, Metered.ca, Xirsys) all have hard monthly limits — typically a few GB.
• Self-hosted TURN (coturn) on a VPS with unmetered bandwidth costs ~$10–30 / month and handles the small-scale alpha population fine.

Decision punted: when this issue is picked up, pick the smallest-viable option (self-hosted coturn on the directory VM, gated to known directory-issued tokens to prevent abuse).

Cross-platform reality

• Linux x86_64 / Windows x86_64: libdatachannel builds cleanly. Tested.
• macOS: libdatachannel works; not currently a build target, but no extra obstacle.
• Linux ARM64 (Pi 4/5): libdatachannel builds. Encoder CPU cost on Pi is the limiting factor, not WebRTC overhead.
• Linux ARM32 (Pi 3): should build but very limited; documented best-effort.

UI surface

• New endpoint mode in the Publish dropdown: WebRTC (peer-to-peer) becomes the new recommended default once stable. Tunnel options remain for fallback / compatibility.
• Client browse list shows a P2P badge on WebRTC entries (vs. the cloudflared / direct icons from feat: public stream directory — opt-in discovery for remote RetroCapture streams #49).
• Optional: surface "connection quality" once connected (bitrate, packet loss, RTT) — WebRTC exposes this via getStats.

Open questions

• Library decision: libdatachannel is the clear front-runner but worth a final eval against the alternatives before committing.
• TURN hosting: self-host vs free-tier-provider vs both with fallback. Tied to who's paying for the directory service.
• Codec configuration: which H.264 profile we emit may need to change to match WebRTC's expectations. May need a MediaEncoder variant for the WebRTC path.
• SFU at some point: do we ever want one? Decision needed before the architecture gets too 1:1-locked.
• Signaling protocol: roll our own JSON over WebSocket vs adopt an existing one (Matrix's, or WHIP/WHEP — these are emerging standards). WHIP/WHEP are particularly interesting because they're HTTP-based and could ride on top of the directory's existing API surface.
• Bandwidth metering / fair use: if we run a TURN server, what stops one user from monopolising it? Probably token-rate-limit per directory-issued streamId.
• Audio sync vs WebRTC's audio jitter buffer: our feat: shader-preserving distributed playback — "Connect to Remote Stream" client mode #47 audio path is finely tuned around the existing MPEG-TS clock. WebRTC owns the audio jitter buffer; some of that tuning has to be re-done or yielded to WebRTC's defaults.

Non-goals

• Replacing the MPEG-TS / HTTP endpoints. They keep existing for direct, no-account use cases and for non-RetroCapture consumers (VLC, mpv, browser portal).
• Multi-party calls / many-to-many topology. Strictly 1:N (one host, many viewers), and N is bounded by host CPU.
• Building an SFU. Acknowledged as the right scaling answer above ~50 concurrent viewers per host but out of scope here.
• Browser-as-viewer support. WebRTC could technically enable that, but the shader pipeline is desktop-native and won't run in a browser without a major rewrite.
• Self-hosting STUN. Use public STUN. Self-host TURN only.

Acceptance criteria

When/if this lands, it should achieve:

• Host can publish with endpointMode: "webrtc" and the entry appears in the directory.
• Client can join a webrtc-mode entry; ICE establishes a direct path when network allows, falls back to TURN otherwise.
• Streams flow peer-to-peer end-to-end encrypted (DTLS-SRTP). Directory verified unable to read frame bytes.
• Works behind strict CGNAT (TURN fallback validated on a real CGNAT'd network — typical Brazilian mobile carrier).
• HTTP /raw + /meta continue to work for non-WebRTC clients on the same host.
• Older RetroCapture builds (pre-WebRTC) show WebRTC entries in the browse list as "unsupported" with a clear tooltip; do not crash.
• Connection-quality stats visible to the user (at minimum: current bitrate, packet loss, RTT).
• TURN bandwidth gated to authenticated directory tokens (no public abuse vector).

Notes

Depends on #49 (the stream directory) existing first — the signaling channel piggybacks on it. Without the directory there's no rendezvous point and WebRTC's first step (find each other) has no answer.

This issue is not on any milestone. It records the architectural direction so future-us (or a contributor) doesn't have to re-do the research before deciding whether to act on it. Probably revisited after #49 has been in real use for some months.