Skip to content

Commit 4dc27ae

Browse files
rodrigokclaude
rodrigok
and
claude
committed
feat(video-conf): blur the background properly, and say what is being sent
Background blur was `@livekit/track-processors`, which composites the background into a texture at a quarter of the frame's size and stretches it back — a 480×270 background on a 1080p frame. That upscale, not the radius, is what looked like low quality, and the factor is a constant in the library: setting blur to 0.1 and still seeing a heavy, coarse background is what found it. So the blur is ours now. MediaPipe's segmenter and a 2D canvas, compositing at full frame size in three draws, with `ctx.filter` doing the Gaussian on the GPU. Strengths are fractions of frame height rather than pixels, since the same track is watched at whatever size the other end's tile happens to be. Changing strength is a number on the running processor, and "no blur" leaves it attached passing frames through, because detaching one re-publishes the camera. Two things about the mask are the opposite of what they look like, and both were wrong first: - MediaPipe returns a mask the size of the image it is given, so segmenting the frame meant reading 1080p worth of mask off the GPU every frame — 94ms. It segments a copy scaled to the model's own input size instead, which is what the model does internally anyway, and the same frame costs 20ms. Segmentation runs on its own 20Hz clock from there. - which category is the person differs by model: the two-class one marks the *room*, the multiclass one marks five kinds of person. Reading it the wrong way blurred the face and left the room sharp. It now builds a lookup table from `getLabels()` — everything not called `background` — which is right for both. The multiclass model is the default: its separate `hair` class holds an edge far better, at 65× the download. Alongside it, the same three choices — noise cancelling, send resolution, blur — are now in the preflight's own mic and camera menus, which meant giving the preflight a real LiveKit track so its preview runs the processor the call will run rather than showing an unblurred picture beside a blurred promise. The local tile carries a badge with the resolution actually going out, read from `getRTCStatsReport()`, because the encoder picks simulcast layers and what leaves the machine is often not what the camera captured. Also fixes the camera flashing whenever the microphone was toggled or switched: the video element was being handed a new `srcObject` on every render that touched audio. `@livekit/track-processors` is dropped; `@mediapipe/tasks-vision` is a direct dependency, pinned because the WASM URL carries its version. Known, and written down in the feature doc: the resolution picker sets a capture constraint rather than an encoder cap, and a remembered blur level is still not applied on join. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 1dce466 commit 4dc27ae

26 files changed

Lines changed: 1434 additions & 147 deletions

‎apps/meteor/client/views/conference/CallDeviceMenu.tsx‎

Lines changed: 65 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,22 @@ import { useTranslation } from 'react-i18next';
77

88
import { useDropdownVisibility } from '../room/Header/Omnichannel/QuickActions/hooks/useDropdownVisibility';
99

10+
type Choice = { id: string; name: string; note?: string };
11+
1012
type CallDeviceMenuProps = {
1113
icon: IconName;
1214
label: string;
13-
devices: MediaDeviceInfo[];
15+
/** The devices to choose between. Ignored when `choices` is given. */
16+
devices?: MediaDeviceInfo[];
1417
selectedId?: string;
1518
onSelect: (deviceId: string) => void;
19+
/**
20+
* Further groups of choices under the devices, in the same dropdown: what to do about noise, how much detail to
21+
* send, how much to blur. They belong to the device they are about — noise is a fact about the microphone, blur
22+
* about the camera — so they live behind the same control rather than in a row of their own, which is also where
23+
* the call itself puts them.
24+
*/
25+
sections?: { title: string; choices: Choice[]; selectedId?: string; onSelect: (id: string) => void }[];
1626
};
1727

1828
/**
@@ -51,18 +61,52 @@ const nameStyles = css`
5161
* Separate from `ui-voip`'s in-call pickers on purpose: those dispatch through the call's own view context to
5262
* switch a device mid-call, and there is no call here yet. This one only records a choice for the join to carry.
5363
*/
54-
const CallDeviceMenu = ({ icon, label, devices, selectedId, onSelect }: CallDeviceMenuProps) => {
64+
const CallDeviceMenu = ({ icon, label, devices, selectedId, onSelect, sections }: CallDeviceMenuProps) => {
5565
const { t } = useTranslation();
5666

5767
const reference = useRef<HTMLButtonElement>(null);
5868
const target = useRef<HTMLElement>(null);
5969
const { isVisible, toggle } = useDropdownVisibility({ reference, target });
6070

61-
// Shared with the in-call pickers, so a device is named and ordered the same way before a call and inside one.
62-
const ordered = useMemo(() => orderDevices(devices), [devices]);
71+
// Devices and plain choices are reduced to the same three fields, so everything below draws one kind of row.
72+
// Devices go through `orderDevices` first, shared with the in-call pickers, so a device is named and ordered the
73+
// same way before a call and inside one.
74+
const rows = useMemo((): Choice[] => {
75+
return orderDevices(devices ?? []).map((device) => ({
76+
id: device.deviceId,
77+
// A device the browser hasn't named yet — permission was granted after it was enumerated.
78+
name: deviceName(device.label),
79+
...(device.deviceId === SYSTEM_DEFAULT_DEVICE_ID && { note: 'system-default' }),
80+
}));
81+
}, [devices]);
6382

64-
const currentId = selectedId ?? ordered[0]?.deviceId;
65-
const current = ordered.find(({ deviceId }) => deviceId === currentId);
83+
const currentId = selectedId ?? rows[0]?.id;
84+
const current = rows.find(({ id }) => id === currentId);
85+
86+
const renderRow = (row: Choice, isCurrent: boolean, choose: () => void) => (
87+
<Option
88+
key={row.id}
89+
selected={isCurrent}
90+
onClick={() => {
91+
choose();
92+
toggle(false);
93+
}}
94+
>
95+
<OptionContent>
96+
<Box withTruncatedText>{row.name || t('Default')}</Box>
97+
{row.note && (
98+
<Box fontScale='c1' color='hint'>
99+
{row.note === 'system-default' ? `${t('System')} ${t('Default').toLowerCase()}` : t(row.note as 'Default')}
100+
</Box>
101+
)}
102+
</OptionContent>
103+
{isCurrent && (
104+
<OptionColumn>
105+
<Icon name='check' size='x20' color='status-font-on-info' />
106+
</OptionColumn>
107+
)}
108+
</Option>
109+
);
66110

67111
return (
68112
<Box display='flex' alignItems='center' minWidth={0}>
@@ -73,41 +117,30 @@ const CallDeviceMenu = ({ icon, label, devices, selectedId, onSelect }: CallDevi
73117
aria-label={label}
74118
aria-haspopup='listbox'
75119
aria-expanded={isVisible}
76-
title={current ? deviceName(current.label) : label}
77-
disabled={!ordered.length}
120+
title={current?.name || label}
121+
disabled={!rows.length}
78122
onClick={() => toggle()}
79123
>
80124
<Icon name={icon} size='x16' flexShrink={0} />
81-
<Box className={nameStyles}>{current ? deviceName(current.label) || t('Default') : label}</Box>
125+
<Box className={nameStyles}>{current?.name || label}</Box>
82126
<Icon name={isVisible ? 'chevron-up' : 'chevron-down'} size='x16' flexShrink={0} />
83127
</Button>
84128

85129
{isVisible && (
86130
<Dropdown reference={reference} ref={target} placement='top-start'>
87-
{ordered.map((device) => (
88-
<Option
89-
key={device.deviceId}
90-
selected={device.deviceId === currentId}
91-
onClick={() => {
92-
onSelect(device.deviceId);
93-
toggle(false);
94-
}}
95-
>
96-
<OptionContent>
97-
{/* A device the browser hasn't named yet — permission was granted after it was enumerated. */}
98-
<Box withTruncatedText>{deviceName(device.label) || t('Default')}</Box>
99-
{device.deviceId === SYSTEM_DEFAULT_DEVICE_ID && (
100-
<Box fontScale='c1' color='hint'>
101-
{t('System')} {t('Default').toLowerCase()}
102-
</Box>
103-
)}
104-
</OptionContent>
105-
{device.deviceId === currentId && (
106-
<OptionColumn>
107-
<Icon name='check' size='x20' color='status-font-on-info' />
108-
</OptionColumn>
131+
{rows.map((row) => renderRow(row, row.id === currentId, () => onSelect(row.id)))}
132+
133+
{sections?.map((section) => (
134+
<Box key={section.title}>
135+
{/* A heading, because a list that runs from microphones straight into "no blur" reads as one
136+
list of increasingly strange devices. */}
137+
<Box paddingInline={16} paddingBlockStart={8} paddingBlockEnd={4} fontScale='micro' color='hint'>
138+
{section.title}
139+
</Box>
140+
{section.choices.map((choice) =>
141+
renderRow(choice, choice.id === (section.selectedId ?? section.choices[0]?.id), () => section.onSelect(choice.id)),
109142
)}
110-
</Option>
143+
</Box>
111144
))}
112145
</Dropdown>
113146
)}

‎apps/meteor/client/views/conference/ConferencePreflight.tsx‎

Lines changed: 103 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,23 @@
11
import type { VideoConferenceCapabilities } from '@rocket.chat/core-typings';
22
import { Box, Button, ButtonGroup, CheckBox, Field, FieldRow, Icon, TextInput } from '@rocket.chat/fuselage';
33
import { useBreakpoints } from '@rocket.chat/fuselage-hooks';
4+
import { VoiceActivity } from '@rocket.chat/ui-voip';
45
import type { ComponentProps } from 'react';
56
import { useCallback, useState } from 'react';
67
import { useTranslation } from 'react-i18next';
78

89
import CallDeviceMenu from './CallDeviceMenu';
910
import CallDeviceToggle from './CallDeviceToggle';
1011
import { useCallDevicePreview } from './hooks/useCallDevicePreview';
11-
import type { CallPreferences } from './hooks/useCallPreferences';
12-
import { useCallPreferences, useCallRingPreference } from './hooks/useCallPreferences';
12+
import type { BlurLevel, CallPreferences, NoiseMethod, VideoQuality } from './hooks/useCallPreferences';
13+
import {
14+
useBackgroundBlurPreference,
15+
useCallPreferences,
16+
useCallRingPreference,
17+
useNoiseSuppressionPreference,
18+
useVideoQualityPreference,
19+
} from './hooks/useCallPreferences';
20+
import { usePreviewVideoTrack } from './hooks/usePreviewVideoTrack';
1321
import CallParticipants from '../../components/CallParticipants';
1422

1523
type ConferencePreflightProps = {
@@ -59,6 +67,34 @@ type ConferencePreflightProps = {
5967
* itself on the other. They are separate questions, and putting the decision under a column of controls made it
6068
* read as the last of them rather than the point of the screen. Narrow viewports stack, preview first.
6169
*/
70+
/**
71+
* The methods offered before a call, weakest first.
72+
*
73+
* Krisp is deliberately absent: whether a workspace may use it is only known by attaching it to a published track
74+
* and seeing whether it turns on, which cannot happen until the call exists. Choosing "best available" is what
75+
* leaving this alone does, and the call's own menu shows Krisp once it has proven itself.
76+
*/
77+
const NOISE_CHOICES: { id: NoiseMethod; label: string; note?: string }[] = [
78+
{ id: 'none', label: 'Noise_cancellation_off' },
79+
{ id: 'browser', label: 'Noise_cancellation_standard' },
80+
{ id: 'rnnoise', label: 'Noise_cancellation_rnnoise', note: 'Noise_cancellation_on_this_device' },
81+
];
82+
83+
const BLUR_CHOICES: { id: BlurLevel; label: string }[] = [
84+
{ id: 'none', label: 'Background_blur_none' },
85+
{ id: 'light', label: 'Background_blur_light' },
86+
{ id: 'medium', label: 'Background_blur_medium' },
87+
{ id: 'strong', label: 'Background_blur_strong' },
88+
];
89+
90+
const QUALITY_CHOICES: { id: VideoQuality; label: string }[] = [
91+
{ id: 'auto', label: 'Video_quality_auto' },
92+
{ id: 'h1080', label: 'Video_quality_1080p' },
93+
{ id: 'h720', label: 'Video_quality_720p' },
94+
{ id: 'h360', label: 'Video_quality_360p' },
95+
{ id: 'h180', label: 'Video_quality_180p' },
96+
];
97+
6298
const ConferencePreflight = ({
6399
name,
64100
action,
@@ -77,21 +113,49 @@ const ConferencePreflight = ({
77113
// every time. What it is *allowed* to do is the room's business, not this preference's — see `canChooseRinging`.
78114
const { ring, toggleRing } = useCallRingPreference();
79115

116+
// Chosen here, applied when the call starts. Both are settings about how this person makes calls, and this is the
117+
// screen where those are set — the same two lists appear in the call itself, reading from the same store.
118+
//
119+
// Blur is deliberately *not* offered here. It is the one setting that would have to be shown to be chosen
120+
// honestly, and the preview is a plain camera stream with no segmenter on it: a level picked here would leave this
121+
// screen sharp and the call blurred, which is the sort of quiet lie the rest of this work has been removing.
122+
const { noiseMethod, selectNoiseMethod } = useNoiseSuppressionPreference();
123+
const { videoQuality, selectVideoQuality } = useVideoQualityPreference();
124+
const { blurLevel, selectBlurLevel } = useBackgroundBlurPreference();
125+
80126
// Only a provider that runs the call in here can be told which devices to use. Offering the choice to one
81127
// that can't would be a promise this screen has no way to keep.
82128
const canChooseDevices = Boolean(capabilities.embedded);
83129

84130
const preview = useCallDevicePreview(canChooseDevices, preferences, devices);
85-
const selfView = canChooseDevices && preferences.cam && !!preview.stream;
86131

87-
// Assigned rather than passed as a prop: `srcObject` is not an attribute, so React cannot set it.
132+
// The camera as a LiveKit track, so the blur chosen below is the blur the call will send — see
133+
// `usePreviewVideoTrack`. The rest of the preview (device lists, the microphone behind the level indicator) still
134+
// comes from the hook above.
135+
const previewVideo = usePreviewVideoTrack(canChooseDevices && preferences.cam, {
136+
deviceId: devices.camId,
137+
quality: videoQuality,
138+
blurLevel,
139+
});
140+
141+
const selfView = canChooseDevices && preferences.cam && !!previewVideo.track;
142+
143+
// Attached by the track rather than by assigning `srcObject`: `attach` is what knows to hand over the *processed*
144+
// track when a processor is running, which is the whole reason the preview is a LiveKit track.
88145
const videoRef = useCallback(
89146
(node: HTMLVideoElement | null) => {
90-
if (node) {
91-
node.srcObject = preview.stream;
147+
const { track } = previewVideo;
148+
if (!node || !track) {
149+
return;
92150
}
151+
152+
track.attach(node);
153+
154+
return () => {
155+
track.detach(node);
156+
};
93157
},
94-
[preview.stream],
158+
[previewVideo.track],
95159
);
96160

97161
// Side by side once there is room for both; stacked below that, with the preview still first.
@@ -164,14 +228,23 @@ const ConferencePreflight = ({
164228
{t('Which_devices_are_used_is_chosen_in_the_call')}
165229
</Box>
166230
)}
167-
{preferences.cam && canChooseDevices && preview.error && (
231+
{preferences.cam && canChooseDevices && (preview.error || previewVideo.error) && (
168232
<Box fontScale='c1' color='hint' marginBlockStart={4} textAlign='center' paddingInline={24}>
169233
{t('Could_not_access_your_camera')}
170234
</Box>
171235
)}
172236
</>
173237
)}
174238

239+
{/* In the corner of the preview, the way every call product shows it: proof before you join that the
240+
microphone is picked up and working, which is the one thing this screen cannot otherwise tell you.
241+
Only while the mic is on — there is nothing to show from a microphone that will not be sent. */}
242+
{canChooseDevices && preferences.mic && preview.stream && (
243+
<Box position='absolute' style={{ bottom: 12, left: 12 }} display='flex'>
244+
<VoiceActivity stream={preview.stream} size={16} badge />
245+
</Box>
246+
)}
247+
175248
{/* Over the preview, where they belong to the thing they change — and where every call UI puts them. */}
176249
<Box position='absolute' style={{ bottom: 12 }} display='flex' justifyContent='center'>
177250
<ButtonGroup>
@@ -219,6 +292,14 @@ const ConferencePreflight = ({
219292
devices={preview.audioInputs}
220293
selectedId={devices.micId}
221294
onSelect={(deviceId) => selectDevice('mic', deviceId)}
295+
sections={[
296+
{
297+
title: t('Noise_cancellation'),
298+
choices: NOISE_CHOICES.map(({ id, label: name, note }) => ({ id, name: t(name), note })),
299+
selectedId: noiseMethod,
300+
onSelect: (method) => selectNoiseMethod(method as NoiseMethod),
301+
},
302+
]}
222303
/>
223304
)}
224305
<CallDeviceMenu
@@ -235,6 +316,20 @@ const ConferencePreflight = ({
235316
devices={preview.videoInputs}
236317
selectedId={devices.camId}
237318
onSelect={(deviceId) => selectDevice('cam', deviceId)}
319+
sections={[
320+
{
321+
title: t('Video_quality'),
322+
choices: QUALITY_CHOICES.map(({ id, label: name }) => ({ id, name: t(name) })),
323+
selectedId: videoQuality,
324+
onSelect: (quality) => selectVideoQuality(quality as VideoQuality),
325+
},
326+
{
327+
title: t('Background_blur'),
328+
choices: BLUR_CHOICES.map(({ id, label: name }) => ({ id, name: t(name) })),
329+
selectedId: blurLevel,
330+
onSelect: (level) => selectBlurLevel(level as BlurLevel),
331+
},
332+
]}
238333
/>
239334
)}
240335
</Box>

‎apps/meteor/client/views/conference/hooks/useCallDevicePreview.ts‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,8 @@ export const useCallDevicePreview = (enabled: boolean, { mic, cam }: CallPrefere
3232
const [devices, setDevices] = useState<MediaDeviceInfo[]>([]);
3333
const [error, setError] = useState(false);
3434

35-
// Nothing is asked of the browser unless something is actually on: opening the camera to preview a camera the
36-
// user turned off would light their webcam for no reason.
35+
// Nothing is asked of the browser unless the microphone is actually on. The camera is not this hook's concern any
36+
// more, but `cam` still matters to the *labels*: permission for either is what puts names on the device lists.
3737
const wanted = enabled && (mic || cam);
3838

3939
// The lists, kept current on their own. `devicechange` covers a headset arriving or leaving mid-decision.
@@ -77,7 +77,9 @@ export const useCallDevicePreview = (enabled: boolean, { mic, cam }: CallPrefere
7777
return deviceId ? { deviceId: { exact: deviceId } } : true;
7878
};
7979

80-
const constraints: MediaStreamConstraints = { audio: wantDevice(mic, micId), video: wantDevice(cam, camId) };
80+
// Audio only. The camera is opened by `usePreviewVideoTrack` as a LiveKit track, because a track is what a blur
81+
// processor can attach to — opening it here as well would light the camera twice for one preview.
82+
const constraints: MediaStreamConstraints = { audio: wantDevice(mic, micId), video: false };
8183

8284
navigator.mediaDevices
8385
.getUserMedia(constraints)

0 commit comments

Comments
 (0)