Log stalled event delivery to the client - #403
Open
vincebln2 wants to merge 3 commits into
Open
Conversation
When the client stops reading its pipe, Send blocks and every later event queues behind it, reaching the client late. That is invisible today: the run just shows flaky waiters or listeners firing after removal. A Send that blocks for a second or more now logs the client, the event method, and the stall duration, so the next incident names the wedged side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #397. Observability for mechanism 1 (events reaching the client late): when the client stops reading its pipe, the router's Send blocks and every later event queues behind it, arriving late. Today that is invisible; the run just shows a flaky waiter or a listener firing after removal.
A Send that blocks for a second or more now logs the client id, the event method, and the stall duration to stderr. Healthy delivery is microseconds, so the log stays quiet outside genuine stalls. With this in place the next incident names the wedged side instead of leaving only a hung test to interpret.
Verified: go build, go vet, and the api package tests pass; the log line only fires above a one second stall, well beyond scheduler jitter on loaded runners.