fix(mdns): release the previous SHIP instance on interface re-announcement - #109
Open
andig wants to merge 3 commits into
Open
fix(mdns): release the previous SHIP instance on interface re-announcement#109andig wants to merge 3 commits into
andig wants to merge 3 commits into
Conversation
|
coverage: 93.096% (+0.004%) from 93.092% — andig:fix/reannounce-entrygroup-leak into enbility:dev |
…ement reannounceWithNewInterfaces cleared isAnnounced up front and then called AnnounceMdnsEntry, which overwrites instanceID without releasing the previous provider instance. Every interface change therefore leaked one provider-side object - an avahi EntryGroup - until the daemon rejected further allocations with "Too many objects", after which service resolution fails permanently. Capture the live instance, announce the new one, then release the old, matching the create-then-swap already used by SetAutoAccept and by the pairing loop in this same function. Clearing isAnnounced is deferred until just before the re-announcement so the no-interfaces path can still tear the service down, and it is restored when the announcement fails and the previous one stays live.
andig
force-pushed
the
fix/reannounce-entrygroup-leak
branch
from
August 31, 2026 16:20
bade5ec to
ed7d718
Compare
This was referenced Aug 31, 2026
kirollosnct
added a commit
that referenced
this pull request
Sep 1, 2026
`go test -race -timeout=90s -count=3 ./ship ./hub` gives **each test binary** a 90s budget. `./hub` needs ~30s per `-race` run, so `-count=3` lands at ~88s locally — inside the budget, with no headroom at all: ``` ok github.com/enbility/ship-go/ship 29.031s ok github.com/enbility/ship-go/hub 88.360s ``` On a GitHub runner it tips over and the job fails with `panic: test timed out after 1m30s`. The dump reads like a deadlock, but the goroutines it names are ordinary: the `running tests:` header shows only `TestHubConnectionsServerSuite` (2s in), and the two live goroutines are a `time.Sleep` inside the test and the 500ms delayed-callback goroutine from `HandleShipHandshakeStateUpdate`. Nothing is stuck — the clock ran out mid-suite. This is not tied to any one branch. The same job has been flapping on `dev` for weeks; run 33413520498 (PR #109, an mdns-only change) is just the latest to hit it. ### Change Raise the timeout on that one step to 300s. Still well inside the job's `timeout-minutes: 15`. The other `-timeout=90s` steps in the file are `-count=1` or `-run`-filtered subsets and have plenty of margin. Not addressed here: `./hub` spending ~13s of its 30s in `TestHandleShipHandshakeStateUpdateTestSuite`, where ~25 tests each wait out the hardcoded 500ms delay in `hub_shipconnection.go`. Making that delay injectable would cut the runtime by a third — worth its own PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Registering a remote service expresses fresh connection intent, but the per-SKI retry backoff (up to 10-20s) and a pending delay timer survived, deferring the attempt far beyond what a caller waiting for the connection expects. Cancel both on register, and cancel the pending timer on unregister as well so it no longer keeps the attempt flag set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
reannounceWithNewInterfacesclearsisAnnouncedup front and then callsAnnounceMdnsEntry, which assigns a freshinstanceIDwithout ever releasing the previous one. Every interface change therefore leaks one provider-side object — for the avahi provider, anEntryGroup.The failure mode is not gradual. Once the leaked objects reach avahi-daemon's
objects-per-client-max(1024 by default), the daemon rejects every further allocation on that D-Bus connection withToo many objects.Server.ResolveServiceneeds an allocation too, so from that point on no discovered service can be resolved any more —processServicereturns the error, nothing retries, and the entry is lost until the client reconnects.I hit exactly this shape in a downstream stack running v0.6.0 (where the leak came from the missing already-announced guard in
AnnounceMdnsEntry, since fixed on dev). In a 10.5 h log, the 1025th announcement was the firstToo many objects, and the next remote restart after that permanently lost the peer from the discovery list even though the SHIP connection stayed up. ThereannounceWithNewInterfacespath is the same bug with a slower trigger: it is gated on real interface appear/disappear rather than on a timer, so it needs a flapping interface, but it converges on the same dead end.Change
Capture the live instance, announce the new one, then release the old — the create-then-swap that
SetAutoAcceptand the pairing loop in this same function already do.Two details fall out of the reordering:
isAnnouncedis deferred until just before the re-announcement. Previously it was cleared first, which made the no-interfaces path'sUnannounceMdnsEntry()return early at its ownisServiceAnnounced()guard — so when every interface disappeared the SHIP service was never torn down at all.Test_reannounceWithNewInterfaces_NoInterfaces_WithPairingdocumented that early return in a comment; the test now asserts the teardown instead.AnnounceMdnsEntryleavesinstanceIDuntouched and the previous announcement is still live, so the flag is restored. Otherwise the nextAnnounceMdnsEntrywould allocate a second instance on top of the first — the same leak by another route.Tests
Two added:
Test_reannounceWithNewInterfaces_ReleasesOldShipInstance(the swap) andTest_reannounceWithNewInterfaces_KeepsStateOnAnnounceFailure(the failure path).go test ./...passes, and the mdns suite is clean under-race.🤖 Generated with Claude Code