fix: buffer writes during disconnect to prevent programs in terminal from freezing#731
Merged
MisterTea merged 2 commits intoMisterTea:masterfrom Apr 1, 2026
Merged
Conversation
94cb593 to
4f2cf10
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #731 +/- ##
==========================================
- Coverage 87.05% 86.96% -0.09%
==========================================
Files 70 70
Lines 5161 5210 +49
Branches 483 487 +4
==========================================
+ Hits 4493 4531 +38
- Misses 667 679 +12
+ Partials 1 0 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
When a client disconnects, writePacket() would previously loop forever waiting for the socket to return, causing the entire pipe chain to back up and freeze programs writing to the terminal (like Claude Code). This change buffers data during disconnect (up to 4MB) and only blocks when the buffer is full. On reconnect, buffered data is replayed via ET existing recovery mechanism. Changes: - Add BUFFERED_ONLY state to BackedWriterWriteState enum - Buffer packets before checking socket state in BackedWriter::write() - Add DISCONNECT_BUFFER_BYTES constant (4MB) to control when to block - Keep MAX_BACKUP_BYTES at 64MB for normal connected operation - Connection::write() treats BUFFERED_ONLY as success - Add unit test for disconnect buffering behavior Behavior: - Disconnected, buffer < 4MB: buffer data, return success (no drop) - Disconnected, buffer >= 4MB: return SKIPPED, writePacket() blocks (no drop) - Connected, buffer > 64MB: trim oldest packets (original ET behavior) This ensures programs do not freeze on short disconnects while still preserving backpressure for extended outages. Data is never dropped when disconnected - it is either buffered or we block.
4f2cf10 to
9f75c17
Compare
Adds test coverage for the MAX_BACKUP_BYTES (64MB) trim behavior when socket is connected. Verifies that: - Old data is trimmed when buffer exceeds 64MB - Recovery from old sequence numbers fails (data trimmed) - Recovery from recent sequence numbers succeeds
a53cc21 to
7814e0c
Compare
Owner
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a client disconnects,
writePacket()would loop forever waiting for the socket, causing the entire pipe chain to back up. This freezes any program writing to the terminal (e.g. Claude Code would no longer make progress).This fix buffers data during disconnect (up to 4MB) before entering the blocking loop. On reconnect, buffered data is replayed via ET's existing recovery mechanism.
Problem
The root cause was in
BackedWriter::write():socketFd < 0, it returnedSKIPPEDimmediately without bufferingConnection::write()returnedfalseonSKIPPEDwritePacket()looped forever with 100ms sleeps waiting for reconnectSolution
BUFFERED_ONLYreturn state for successful buffering without socketDISCONNECT_BUFFER_BYTESconstant (4MB) to control when to start blockingMAX_BACKUP_BYTESat 64MB for normal connected operationBehavior
The blocking loop still exists but now only triggers after 4MB is buffered. This preserves data for replay on reconnect while still providing backpressure for extended outages.
Changes
BackedWriter.hpp: AddBUFFERED_ONLYenum, addDISCONNECT_BUFFER_BYTESconstantBackedWriter.cpp: Buffer before socket check, respect disconnect limitConnection.cpp: HandleBUFFERED_ONLYas successBackedIOTest.cpp: Update existing test, add new disconnect buffer testTest plan
Evidence from production logs
After deploying this fix, the server logs show successful buffering during disconnects:
These writes completed successfully (returned
BUFFERED_ONLY) instead of blocking. The terminal remained responsive during the disconnect, and output was replayed on reconnect.