Skip to content

Commit 73a6f9d

Browse files
rodrigokclaude
andcommitted
feat(video-conf): ship MediaPipe assets locally for airgapped support
The WASM runtime and TFLite segmentation models were loaded from cdn.jsdelivr.net and storage.googleapis.com, which broke on cloud deployments (CSP blocks the CDN) and on airgapped workspaces (no internet). Ship them from public/mediapipe/ instead, following the same pattern as the RNNoise assets in public/noise-suppressor/. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a547401 commit 73a6f9d

8 files changed

Lines changed: 49 additions & 10 deletions

File tree

apps/meteor/client/views/videoConference/livekit/backgroundBlurProcessor.ts

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,15 @@ import { supportsBackgroundBlur } from './backgroundBlurSupport';
1717
* **15.6 MB** against 244 KB, and roughly twice the work per frame.
1818
* - **selfie** is one class at 256×144 — the landscape shape a call actually is, cheap, and blunter around hair.
1919
*
20-
* Both are fetched from Google's CDN on first use, so a workspace with no way out to the internet gets a caught
21-
* failure and no blur. Serving them from `public/`, as the RNNoise assets are, is what would fix that.
20+
* Both are served from `public/mediapipe/`, alongside the WASM runtime, so airgapped workspaces work out of the box.
2221
*/
2322
export const SEGMENTER_MODELS = {
2423
multiclass: {
25-
url: 'https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_multiclass_256x256/float32/latest/selfie_multiclass_256x256.tflite',
24+
url: '/mediapipe/selfie_multiclass_256x256.tflite',
2625
input: { width: 256, height: 256 },
2726
},
2827
selfie: {
29-
url: 'https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter_landscape/float16/latest/selfie_segmenter_landscape.tflite',
28+
url: '/mediapipe/selfie_segmenter_landscape.tflite',
3029
input: { width: 256, height: 144 },
3130
},
3231
} as const;
@@ -38,10 +37,10 @@ export const SEGMENTER = SEGMENTER_MODELS.multiclass;
3837
* MediaPipe's WASM, which has to match the version of `@mediapipe/tasks-vision` this app depends on — it is the
3938
* runtime for the JS in the package, not an independent thing. Keep the two in step when the package moves.
4039
*
41-
* Both this and the model come from a CDN, so a workspace with no way out to the internet gets a caught failure and
42-
* no blur. Serving them from `public/`, as the RNNoise assets are, is what would fix that.
40+
* Served from `public/mediapipe/wasm/`, copied from the npm package at build time. When `@mediapipe/tasks-vision`
41+
* is updated, re-copy the wasm directory contents.
4342
*/
44-
export const SEGMENTER_WASM = 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.14/wasm';
43+
export const SEGMENTER_WASM = '/mediapipe/wasm';
4544

4645
/**
4746
* Which confidence mask describes the person, and whether it has to be read inside out.
15.6 MB
Binary file not shown.
244 KB
Binary file not shown.

apps/meteor/public/mediapipe/wasm/vision_wasm_internal.js

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
8.99 MB
Binary file not shown.

apps/meteor/public/mediapipe/wasm/vision_wasm_nosimd_internal.js

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Binary file not shown.

docs/features/native-video-conference/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -228,9 +228,9 @@ Two ways of doing it, and which one runs is whichever can — the same arrangeme
228228
- **The camera's own**, via the `backgroundBlur` constraint. Free: the platform does it before the frames reach us.
229229
Some platforms let the app control it; others expose an OS-controlled effect that the app can only observe.
230230
- **Ours**, `BackgroundBlurProcessor` in `apps/meteor/client/views/videoConference/livekit/`: MediaPipe segmentation
231-
and a WebGL2 compositor, as a LiveKit `TrackProcessor`. Works anywhere with the modern APIs, and it is not free — it fetches
232-
MediaPipe's WASM and a model from a CDN (`cdn.jsdelivr.net`, `storage.googleapis.com`) the first time a level is
233-
picked. A workspace with no way out to the internet gets the caught failure and blur stays off.
231+
and a WebGL2 compositor, as a LiveKit `TrackProcessor`. Works anywhere with the modern APIs, and it is not free — it loads
232+
MediaPipe's WASM runtime and a TFLite model from `public/mediapipe/` the first time a level is picked. Both are
233+
shipped with the application, so airgapped workspaces work out of the box.
234234

235235
**Ask `getCapabilities()`, never `applyConstraints`.** `applyConstraints({ backgroundBlur: true })` *resolves
236236
happily* on a browser that has never heard of the constraint — an unrecognised non-required constraint is dropped

0 commit comments

Comments
 (0)