tray: a live core with a saturated listen backlog can be launched over, producing doomed spawns and a misleading error

[Dumbris]

Found by cross-model review (round 6) of #927, which introduced the stale-socket escalation. Not a merge blocker for #927 — the single-writer invariant still holds — but the failure mode is user-visible and confusing.

What happens

onlyEverRefused is used as evidence that a socket file is stale (refusals > 0 and nothing ever accepted during the episode). That is sound for a crashed core, but a live core whose listen backlog is saturated before the episode begins refuses every probe too. Since it never accepts once, the evidence never records somethingAccepted, and after the 60s deadline reportUnresponsiveCore escalates to a launch (CoreProcessManager.swift:792). The escalation's own confirmation and the launch preflight also only ever see refusals, so neither vetoes it.

Consequence

The tray launches cores that are doomed: each new core blocks ~10s on bbolt's exclusive lock, fails, and the ladder retries. The attach watch has already been cancelled. The user eventually sees a retry-exhaustion / database-locked error about a core that is actually alive and healthy — just briefly saturated.

Data integrity is fine throughout: bbolt's exclusive flock is acquired before socket cleanup, so there is never a second writer.

Why the existing test does not cover it

CoreLivenessTests.swift:861 (testACoreThatAcceptedEarlierInTheWindowIsNeverEscalatedOver) models a core that accepts early and then refuses — but it stops the listener entirely after that acceptance, which makes the socket genuinely stale. The uncovered case is a listener that is live for the whole episode and refuses every probe from the start.

Possible directions

• Treat a socket file whose inode/mtime is changing, or which a process still holds open, as evidence against staleness (lsof-equivalent via proc_pidfdinfo, or check the file is still bound).
• Have the spawned core distinguish "database locked" from "failed to start" and, on lock contention, tell the tray to go back to attaching rather than climbing the retry ladder.
• Cap escalation at one attempt per episode, so the worst case is one doomed spawn rather than a ladder.

[Dumbris]

Reproduced live, with two corrections to the severity assessment

Faithfully reproduced on real processes during QA of #927: a real core holding the bbolt lock, SIGSTOPped so it stops accepting, backlog filled until connect #128 returned ECONNREFUSED (kern.ipc.somaxconn=128). The tray waited 61.5s, concluded "stale", and launched four doomed cores in 39 seconds, each dying with "Database is locked by another process" (retries 1/3, 2/3, 3/3).

(a) The blast radius is smaller than this issue assumed

No spawned core ever removed the live core's socket. bbolt takes the data-directory lock before the listener is created, so each doomed core died before reaching cleanupStaleSocket. The socket inode never changed, and after SIGCONT the original core answered {"ready":true} normally. The live core is never stranded and no data is at risk — the single-writer invariant holds with room to spare.

(b) The user-facing symptom is worse than "doomed spawns"

This is the part worth fixing. The tray ends at:

Maximum reconnection attempts exceeded — MCPProxy could not reconnect after multiple attempts. Try restarting from the menu.

Two problems with that end state:

1. The hint points at a control that does not exist. In that state the menu has no Retry item and no Start Core item — only Quit.
2. It is permanent. The core was allowed to become fully healthy again and was answering /ready continuously for a further 75 seconds; the tray stayed wedged on the error the whole time and never re-attached.

So the real cost to a user is a dead-ended tray requiring a manual quit and relaunch, not merely some wasted spawn attempts. Neither behaviour is introduced by #927 — the ladder-exhaustion end state predates it — but the escalation makes this path reachable from a saturated-but-healthy core, which is what ties it to this issue.

Implication for the fix

Whatever is done about onlyEverRefused, the terminal state deserves separate attention: after reconnection attempts are exhausted, the attach watch should keep probing and re-attach when a core answers again, and the error hint should not reference a menu item that is absent.