Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/close_peer_connections_before_signal_teardown.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
livekit: patch
livekit-ffi: patch
---

# Close peer connections before awaiting signal teardown

`SessionInner::close` released the peer connections only after two awaits that can block
indefinitely, so cancelling `close()` — for example by wrapping it in a timeout — left the
transports open and their ICE UDP sockets bound for the lifetime of the process. Long-lived
clients eventually exhausted their file descriptors. The transports are now closed before
the first await, which makes the teardown safe to cancel.
8 changes: 8 additions & 0 deletions livekit/src/room/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -908,6 +908,14 @@ impl Room {
self.inner.rtc_engine.drop_disconnected_updates(enabled);
}

/// Test-only: the publisher transport's current connection state. Lets a test assert
/// that teardown really closed the transport, rather than inferring it from room-level
/// state that can reach `Disconnected` while the transport is still open.
#[cfg(feature = "__lk-e2e-test")]
pub fn publisher_connection_state(&self) -> libwebrtc::prelude::PeerConnectionState {
self.inner.rtc_engine.session().publisher_connection_state()
}

pub async fn get_stats(&self) -> EngineResult<SessionStats> {
self.inner.rtc_engine.get_stats().await
}
Expand Down
20 changes: 16 additions & 4 deletions livekit/src/rtc_engine/rtc_session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -845,6 +845,13 @@ impl RtcSession {
self.inner.drop_disconnected_updates.store(enabled, Ordering::Release);
}

/// Test-only: the publisher transport's current connection state, so tests can assert
/// that teardown actually closed it rather than inferring it from room-level state.
#[cfg(feature = "__lk-e2e-test")]
pub fn publisher_connection_state(&self) -> PeerConnectionState {
self.inner.publisher_pc.peer_connection().connection_state()
}

pub async fn wait_pc_connection(&self) -> EngineResult<()> {
self.inner.wait_pc_connection().await
}
Expand Down Expand Up @@ -1981,6 +1988,15 @@ impl SessionInner {
self.closed.store(true, Ordering::Release);
self.pc_state_notify.notify_waiters();

// Both awaits below are unbounded, and `PeerTransport::close` is synchronous, so
// closing here — before the future can suspend — is what stops a cancelled
// `close()` leaving the ICE sockets bound for the process's lifetime. The
// signalling socket is unaffected, so the Leave below still goes out.
self.publisher_pc.close();
if let Some(ref sub_pc) = self.subscriber_pc {
sub_pc.close();
}
Comment on lines +1991 to +1998

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Cancelling a room shutdown still leaves the media connections open

The media transports are still shut down (self.publisher_pc.close() at livekit/src/rtc_engine/rtc_session.rs:1995) only after the shutdown has already waited on four background workers, so abandoning a shutdown early still leaves them open.
Impact: A caller who bounds room teardown with a short timeout still leaks the room's network sockets for the lifetime of the process, and the new test that asserts otherwise fails.

Why the reorder does not reach the first suspension point

Room::closeRoomSession::close (livekit/src/room/mod.rs:1175-1183) → RtcEngine::closeEngineInner::close (livekit/src/rtc_engine/mod.rs:1052-1065) → RtcSession::close (livekit/src/rtc_engine/rtc_session.rs:797-811). RtcSession::close awaits handle.rtc_task, handle.signal_task, handle.dc_task and handle.dt_sender_task before calling self.inner.close(reason).await, which is where the new publisher_pc.close() lives. Those JoinHandle awaits return Pending on the first poll (the tasks have only just been signalled via close_tx, and on the default current-thread test runtime they cannot even run while the close future is being polled), and the workers themselves contain unbounded awaits.

Consequently the future's first suspension happens at handle.rtc_task.await, still with both peer connections open. livekit/tests/room_test.rs:114-122 cancels at exactly that first suspension (timeout(Duration::ZERO, room.close())) and then asserts PeerConnectionState::Closed, which cannot hold under this call chain.

To make cancellation actually safe, the transport close needs to happen before the task joins in RtcSession::close (or at the very top of the teardown chain), not merely before the signal-client awaits in SessionInner::close.

Prompt for agents
The PR moves publisher/subscriber PeerConnection close to the top of SessionInner::close so that a cancelled close() (e.g. wrapped in a timeout) still releases ICE sockets. However, SessionInner::close is only reached after RtcSession::close (livekit/src/rtc_engine/rtc_session.rs, around lines 797-811) awaits four JoinHandles (rtc_task, signal_task, dc_task, dt_sender_task), and EngineInner::close/RoomSession::close sit above that. Those joins are the real first suspension points of the teardown future, so a cancelled close still leaves the transports open — and the new test test_cancelled_close_still_closes_peer_connections in livekit/tests/room_test.rs, which cancels at the first suspension, should fail. Consider closing the transports synchronously before the task joins (e.g. at the start of RtcSession::close, keeping SessionInner::close idempotent) or otherwise ensuring the synchronous PeerTransport::close runs before any await in the whole teardown chain, and re-verify the test.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


self.signal_client
.send(proto::signal_request::Message::Leave(proto::LeaveRequest {
action: proto::leave_request::Action::Disconnect.into(),
Expand All @@ -1990,10 +2006,6 @@ impl SessionInner {
.await;

self.signal_client.close().await;
self.publisher_pc.close();
if let Some(ref sub_pc) = self.subscriber_pc {
sub_pc.close();
}
}

async fn simulate_scenario(self: &Arc<Self>, scenario: SimulateScenario) -> EngineResult<()> {
Expand Down
33 changes: 33 additions & 0 deletions livekit/tests/room_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ use {
anyhow::{Ok, Result},
chrono::{TimeDelta, TimeZone, Utc},
common::test_rooms,
libwebrtc::prelude::PeerConnectionState,
livekit::{ConnectionState, ParticipantKind, RoomEvent},
std::time::Duration,
tokio::time::{self, timeout},
Expand Down Expand Up @@ -89,3 +90,35 @@ async fn test_participant_disconnect() -> Result<()> {
timeout(Duration::from_secs(15), wait_for_disconnected).await??;
Ok(())
}

/// A cancelled `close()` must still have closed the peer connections.
///
/// `SessionInner::close` ends with two unbounded awaits, so a caller bounding it with a
/// timeout used to drop the future before the transports were closed — and nothing else
/// closes them, so their ICE sockets stayed bound for the process's lifetime. Cancelling at
/// the first `.await` makes this deterministic: the assertion holds only if the close ran
/// before the future suspended.
#[cfg(feature = "__lk-e2e-test")]
#[test_log::test(tokio::test)]
async fn test_cancelled_close_still_closes_peer_connections() -> Result<()> {
let (room, _) = test_rooms(1).await?.pop().unwrap();
assert_eq!(
room.publisher_connection_state(),
PeerConnectionState::Connected,
"publisher should be connected before teardown, or the assertion below proves nothing"
);

// Poll the close exactly once, then drop it. `Duration::ZERO` elapses on the first poll,
// so `timeout` yields `Err` the moment the inner future suspends — cancelling it at the
// earliest possible point rather than at some arbitrary later one.
let cancelled = timeout(Duration::ZERO, room.close()).await;
assert!(cancelled.is_err(), "close should have been cancelled at its first suspension point");

assert_eq!(
room.publisher_connection_state(),
PeerConnectionState::Closed,
"a cancelled close must not leave the publisher transport open: its ICE sockets are \
only released by PeerConnection::close, so anything else leaks them permanently"
);
Ok(())
}
Loading