fix(unix): split oversized GSO sends into smaller chunks by kixelated · Pull Request #2653 · quinn-rs/quinn

kixelated · 2026-05-24T23:06:27Z

(summary is AI generated)

When a Transmit exceeds what one sendmsg syscall can accept — either because it has more segments than max_gso_segments allows, or because its total size exceeds the kernel's per-call limit (u16::MAX bytes) — split it upfront into smaller batches dispatched via separate sendmsg calls. This catches three cases that today silently drop datagrams or rely on the QUIC layer's loss timers to recover:

Stale in-flight GSO transmits left over from a GSO halt (udp: retry failed GSO sendmsg on EIO without GSO #2399). After EIO halts GSO via max_gso_segments.store(1), any transmits queued while GSO was still enabled now violate the new limit and need to be re-batched as individual sends rather than dropped.
The first GSO send that triggers the halt itself (udp: retry failed GSO sendmsg on EIO without GSO #2399). The EIO arm now re-dispatches the failing transmit through send after the halt, where the upfront chunking check splits it into individual datagrams without waiting for a PTO.
Callers that pass oversized Transmit::contents (partial Should quinn-udp automatically chunk Transmit::contents? #2201) — e.g. >65535 bytes total, which the kernel rejects with EMSGSIZE. These are now broken into kernel-acceptable batches, and each chunk still uses GSO when its segment count is > 1.

The public contract of max_gso_segments is unchanged — callers should still query it and size their Transmits accordingly. This path is a defensive net so that callers which don't (or which raced a GSO halt) don't silently drop datagrams.

Cellular Android, where the kernel reports GSO support but the radio driver returns EIO on the first real send, is the original motivation: recovery now completes inside the sendmsg call instead of waiting for a ~1s handshake PTO. Firefox is hot-fixing the same case in D265812 and Firezone hit it independently (firezone/firezone#10279).

Fixes #2399.
Refs #2201.

thomaseizinger · 2026-05-25T05:06:14Z

(untested, and mostly AI generated)

FTR: I don't mind this in principle and I see you've marked it as draft for now. Let us know when you've reviewed it yourself and done some testing on it.

I am glad this is being worked on. We do this on top of quinn-udp right now. I suggest you have a look at https://github.com/firezone/firezone/blob/445f46da6273dcce917a197f0747f91af782624b/rust/libs/connlib/socket-factory/src/lib.rs#L412 too.

flub · 2026-06-03T07:30:30Z

+    state: &UdpSocketState,
+    io: SockRef<'_>,
+    transmit: &Transmit<'_>,
+    max_per_chunk: usize,


is there any practical value to making this dynamic rather than simply hardcoding this to 1?

Yeah it's hard to follow, working on renaming stuff and avoiding some recursion now.

max_per_chunk should be called gso_segments or something. It's the maximum number of segments that can be contained in a single packet, initially starting at >1 (depending on the OS) and shrinking down to 1 on an error.

If it shrinks down to 1, we call send again to repeat this logic, but gso_segments will be 1 now.

flub · 2026-06-03T07:31:47Z

+) -> io::Result<()> {
+    let segment_size = transmit
+        .effective_segment_size()
+        .expect("caller verified segment_size is set");


If there isn't, can this not safely create a single chunk and send it instead of having to panic? The invariants being held so far from each other often ends up in tears sometimes. I feel like we need to try harder to avoid these things.

This should be passed in as an argument instead of recomputing it anyway.

kixelated · 2026-06-03T20:14:04Z

kk I simplified the logic and removed the recursion. It's still untested (on my Mac and the customer has the failing Android anyway).

Some of the comments are still AI generated, but I've closely reviewed everything.

kixelated · 2026-06-03T20:15:42Z

+        // `send_batched` halting GSO mid-syscall.
+        if e.raw_os_error() == Some(libc::EINVAL)
+            && !state.sendmsg_einval()
+            && transmit.effective_segment_size().is_none()


This fixes an existing bug, where a GSO failure could also disable ECN. Can split into a separate PR if you'd prefer.

thomaseizinger

Left two initial comments.

I also think we should have integration-level test coverage for this.

thomaseizinger · 2026-06-03T20:17:53Z

+    #[cfg(any(target_os = "linux", target_os = "android"))]
+    if let Some(segment_size) = transmit.effective_segment_size() {
+        // Get the (GSO) segment size, and compute the number of segments we could fit into a UDP payload.
+        let segment_count = max_segments_per_send(state.max_gso_segments(), segment_size);
+        return send_batched(state, io, transmit, segment_size, segment_count);
+    }
+
+    // Fall back to sending each packet individually.
+    send_packet(state, &io, transmit)


I don't really understand this. There should not be a need for branching here because a single packet is a batch of size one with segment_size == packet_len, right?

It's mostly a micro-optimization to avoid running the send_batched loop/logic if GSO isn't enabled/supported.

You're right that we could effective_segment_size instead of transmit.segment_size. It would end up being a batch of 1 packet.

Lemme see if I can combine the code paths to avoid the branching.

thomaseizinger · 2026-06-03T20:19:41Z

+/// The kernel rejects any GSO send whose total `contents` length exceeds
+/// `u16::MAX`, regardless of `max_gso_segments`.
+#[cfg(any(target_os = "linux", target_os = "android", test))]
+const MAX_SEND_BYTES: usize = u16::MAX as usize;


As I've already pointed out, it is not that easy. You need to take into account IP and UDP headers here. See the link I've left in the previous comment.

kixelated · 2026-06-03T22:37:59Z

@thomaseizinger better?

thomaseizinger

Overall better yes but I am not sure we are going into the right direction in general. Some questions to think about:

Shouldn't this logic apply to all code-paths, i.e. Windows + Apple as well? The way I implemented this in Firezone, it is running above quinn-udp, regardless of whether GSO / apple-fast path is active.
What about integration-level tests? Whether or not we are doing the slicing correctly is only surfaced if we actually hit a kernel and pass it a giant Transmit, i.e. one with a content that is larger than u16::MAX.

If we don't apply this loop globally, then I think we should inline this into the current send function. We already have a loop there and we just need to keep looping and call prepare_msg, right?

kixelated · 2026-06-24T19:59:20Z

Shouldn't this logic apply to all code-paths, i.e. Windows + Apple as well? The way I implemented this in Firezone, it is running above quinn-udp, regardless of whether GSO / apple-fast path is active.

I had Claude whip up something based on what I think you're suggested. But I don't really understand the suggestion and I don't see the purpose, except to increase the scope?

AFAIK sendmsg_x is sendmmsg, not sendmsg + GSO. The batching logic is completely different?

thomaseizinger · 2026-06-24T21:19:31Z

AFAIK sendmsg_x is sendmmsg, not sendmsg + GSO. The batching logic is completely different?

The mechanism is different but the API exposed by quinn-udp is the same: Take in a giant Transmit and send it with as little syscall overhead as possible.

For connected UDP sockets, sendmsg_x has a fast-path in the XNU kernel where the buffers get fast tracked through the layers so it is not quite the same as sendmmsg on Linux.

In both cases though, the provided Transmit already needs to be appropriately sized according to the current GSO configuration (i.e. batch-size). Passing to big of a buffer, i.e. not doing the slicing, will result in IO errors.

kixelated · 2026-07-07T20:26:35Z

@thomaseizinger so I tested on Windows and it doesn't have the same u16::MAX limitation. IDK about Apple.

My main goal is to retry failed packets when disabling GSO. Right now WebSocket is winning the race with QUIC because it takes 500ms+ (or whatever) to retransmit the dropped Initial packets. The current behavior really hurts connection time for Android.

IDK I can make a smaller PR that doesn't touch the batch size if you want. I don't really want to get sucked into scope creep if it makes it harder to land a fix.

When a `Transmit` exceeds what one `sendmsg` syscall can accept — either because it has more segments than `max_gso_segments` allows, or because its total size exceeds the kernel's per-call limit (`u16::MAX` bytes) — split it upfront into smaller batches dispatched via separate `sendmsg` calls. This catches three cases that today silently drop datagrams or rely on the QUIC layer's loss timers to recover: * Stale in-flight GSO transmits left over from a GSO halt (quinn-rs#2399). After `EIO` halts GSO via `max_gso_segments.store(1)`, any transmits queued while GSO was still enabled now violate the new limit and need to be re-batched as individual sends rather than dropped. * The first GSO send that triggers the halt itself (quinn-rs#2399). The `EIO` arm now re-dispatches the failing transmit through `send` after the halt, where the upfront chunking check splits it into individual datagrams without waiting for a PTO. * Callers that pass oversized `Transmit::contents` (partial quinn-rs#2201) — e.g. >65535 bytes total, which the kernel rejects with `EMSGSIZE`. These are now broken into kernel-acceptable batches, and each chunk still uses GSO when its segment count is > 1. Each intermediate batch is rounded down to a whole number of segments (`next_batch_bytes`): the `u16::MAX` byte cap is rarely a multiple of `segment_size`, and the kernel segments a GSO buffer purely by byte offset, so splitting at the raw cap would emit a short datagram in the middle of the stream and shift every segment after it. Only the genuine final batch keeps a partial last segment. The public contract of `max_gso_segments` is unchanged — callers should still query it and size their `Transmit`s accordingly. This path is a defensive net so that callers which don't (or which raced a GSO halt) don't silently drop datagrams. Cellular Android, where the kernel reports GSO support but the radio driver returns `EIO` on the first real send, is the original motivation: recovery now completes inside the `sendmsg` call instead of waiting for a ~1s handshake PTO. Firefox is hot-fixing the same case in D265812 and Firezone hit it independently (firezone/firezone#10279). Adds `gso_oversized_transmit_is_sliced`, which sends a `>u16::MAX` GSO transmit through a real socket and verifies every datagram arrives intact — the slicing correctness only surfaces once a kernel actually segments the buffer. Unit tests cover the `max_batch_bytes` / `next_batch_bytes` arithmetic on all platforms. Fixes quinn-rs#2399. Refs quinn-rs#2201. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

thomaseizinger · 2026-07-08T01:17:53Z

Right now WebSocket is winning the race with QUIC because it takes 500ms+ (or whatever) to retransmit the dropped Initial packets. The current behavior really hurts connection time for Android.

I am only a maintainer of quinn-udp so I can't speak for quinn but an easy, targeted fix should be possible inside quinn itself the same way we currently immediately retry this in Firezone without waiting for a retransmit.

kixelated · 2026-07-09T19:59:00Z

Right now WebSocket is winning the race with QUIC because it takes 500ms+ (or whatever) to retransmit the dropped Initial packets. The current behavior really hurts connection time for Android.

I am only a maintainer of quinn-udp so I can't speak for quinn but an easy, targeted fix should be possible inside quinn itself the same way we currently immediately retry this in Firezone without waiting for a retransmit.

I don't think so, because you have to know if the EIO error occurred while GSO was enabled? Otherwise you would retry on EIO errors after GSO was disabled?

And I mean the layering just seems gross. Like you'd have to know you're supposed to retry specific errors... the library itself should just retry.

kixelated · 2026-07-09T21:07:44Z

I made #2724 as the "simpler and actually fixes my bug" PR. It's still more correct to split oversized GSO sends but it's a separate issue.

flub reviewed Jun 3, 2026

View reviewed changes

kixelated force-pushed the udp-retry-gso-eio branch 2 times, most recently from e7cc1fb to ae31e3a Compare June 3, 2026 20:13

kixelated commented Jun 3, 2026

View reviewed changes

thomaseizinger requested changes Jun 3, 2026

View reviewed changes

kixelated force-pushed the udp-retry-gso-eio branch 2 times, most recently from 36af237 to 26ef777 Compare June 3, 2026 22:35

kixelated force-pushed the udp-retry-gso-eio branch from 26ef777 to a24b09a Compare June 4, 2026 03:48

thomaseizinger reviewed Jun 4, 2026

View reviewed changes

kixelated mentioned this pull request Jun 24, 2026

Explore: hoist GSO chunking into a shared cross-platform loop kixelated/quinn#1

Draft

kixelated force-pushed the udp-retry-gso-eio branch from a24b09a to 1d363b1 Compare June 24, 2026 20:39

aajaces mentioned this pull request Jul 7, 2026

udp: retry failed GSO sendmsg on EIO without GSO #2399

Open

kixelated force-pushed the udp-retry-gso-eio branch from 865aca5 to 14282c0 Compare July 7, 2026 20:43

kixelated marked this pull request as ready for review July 7, 2026 20:46

kixelated requested review from Ralith, djc, gretchenfrage and mxinden as code owners July 7, 2026 20:46

kixelated mentioned this pull request Jul 9, 2026

[noq-udp] Retransmit failing GSO packets and split batches n0-computer/noq#746

Open

Uh oh!

Uh oh!

Conversation

kixelated commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomaseizinger commented May 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kixelated commented Jun 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thomaseizinger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kixelated commented Jun 3, 2026

Uh oh!

thomaseizinger left a comment

Choose a reason for hiding this comment

Uh oh!

kixelated commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomaseizinger commented Jun 24, 2026

Uh oh!

kixelated commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomaseizinger commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kixelated commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kixelated commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kixelated commented May 24, 2026 •

edited

Loading

kixelated commented Jun 24, 2026 •

edited

Loading

kixelated commented Jul 7, 2026 •

edited

Loading

thomaseizinger commented Jul 8, 2026 •

edited

Loading

kixelated commented Jul 9, 2026 •

edited

Loading