perf(live): cut the per-poll socket, refetch and transcription CPU storms

[tanvishdesai]

Three hot paths in a live session were doing work proportional to the whole room to produce a result one client reads, and the transcription service was oversubscribing a 2-core box. Everything here is a do less work change — no new dependencies, no schema changes, no rewrite.

Every number below was measured against a real server with real clients, before and after, not estimated.

1. Joining a room was O(N²)

Every join was announced to the entire room, so the k-th of N students triggered k deliveries. With 200 real socket.io clients:

	frames	per student
before	34,619	200
after	400	1

The teacher is the only consumer of the participant count; the joiner needs its own ack. Both now go where they're read, via a per-room :staff channel.

This also fixes two latent bugs: the leaver never received their own room:left (they'd already left the channel), while every other student did — and cleared their socket-store room state on a stranger's departure.

2. counts:updated was broadcast to 700 and read by 1

It also re-grouped every response in the room on each 1.5s tick, re-deriving counts for polls closed long ago — a scan that grows for the whole session.

Now it counts only the poll whose tally moved, as an indexed count rather than an aggregation, and goes to the staff channel. Measured: 50 student frames → 0, identical values on the teacher side.

Payload is partial, so the client merges rather than replaces. Throttle key is per (room, question) so two polls receiving answers during the grace window can't suppress each other. Scoping by the answered question rather than the room's current one is what makes grace-window stragglers land on the right poll.

3. The poll-transition refetch storm

Every student re-fetched their whole question list when their answering timer expired. Timers are started by a broadcast, so the whole room fired at once, unspread.

For a student who already answered, that request returns exactly what they have — the poll is still the room's live one at expiry, so the answer stays withheld until the next launch supersedes it. Answerers now skip it. Students who missed the poll still fetch, jittered, since that's their only path to seeing the question.

700 students, one poll transition:

	requests	payload	connect refusals	p95
before	700	2,089 KB	1,163	2,513 ms
after	105	310 KB	0	12 ms

The refusals are the important number. 700 simultaneous connections overflow Node's default TCP accept backlog, so requests are rejected at the socket layer before reaching the app and have to be retried. That's a concrete mechanism for the "lags or doesn't work properly around 700 students" symptom, and it's invisible to any application-level metric.

Both halves matter: skipping alone is 7.4×, the jitter is a further 25×.

4. Transcription CPU

cpu_threads was unset and beam_size=5. On 2 vCPUs, CTranslate2's default spins up more OpenMP threads than there are cores — while Node, MongoDB and Redis compete for the same two, including during an answer burst (recording resumes as soon as a poll launches).

Wall time was never the constraint; a 10s window only has to finish in well under 10s. CPU-seconds taken from the event loop is what matters. Pinned to two logical cores to match the droplet, 10s of lecture speech, mean of 5 runs:

threads	beam	wall	CPU-s
default	5	2.710s	5.241
2	5	1.887s	3.744
1	1	1.951s	1.938

2.70× less CPU and 28% less wall time. The default is worse than an explicit threads=2 on both axes for identical work — oversubscription overhead that only appears on a small box. An earlier 16-core run made the pin look like a 28% latency cost rather than a gain, which is why this was re-measured under droplet conditions.

Transcript output is byte-identical across all eight thread/beam combinations tested, so this is not an accuracy trade. beam_size=1 is the smaller half (~1.2×); the thread pin is the rest. Both are env-overridable (TRANSCRIPTION_CPU_THREADS, TRANSCRIPTION_BEAM_SIZE).

Tests

Two new suites, 11 tests, alongside the existing ones (all green):

• liveCounts.test.js — replaying the partial broadcasts through the client's merge always reproduces the old full-room aggregation, across multiple polls, cross-room ids, and the grace-window straggler case. Plus an explain() assertion that the scoped filter stays IXSCAN.
• liveResponsesReveal.test.js — asserts against the real route the premise that skipping the answerer's refetch rests on: the skipped request returns a byte-identical body, the answer stays withheld while the poll is live, the reveal fires on supersede, non-answerers still need the fetch, and membership is still enforced after the switch to the cached check.

Not verified

• Multi-instance. All E2E ran single-instance (no REDIS_URL). ${roomCode}:staff is an ordinary socket.io room and the Redis adapter treats it identically to roomCode — the same mechanism already in production — but it wasn't exercised. Worth one check on staging with both pm2 instances up: the teacher's answer badge and participant counter should still update.
• Latencies were measured on a 16-core dev box with a local Mongo. Absolute numbers will be worse on the droplet; the ratios are what transfer.

Deploy note

No migration and no required config. The two TRANSCRIPTION_* vars take effect on the next restart of spandan-transcription.service with no action needed; documented as commented overrides in the unit file.

🤖 Generated with Claude Code