Make coordinator lifecycle event-driven and restart-durable

[e-jung]

Summary

Orca persists orchestration tasks, dispatches, and messages, but the coordinator run loop is still poll-driven and keeps important campaign state in memory. A successful worker_done, escalation, or decision gate may remain unseen until an explicit inbox check or the next coordinator poll; if the coordinator/runtime restarts, an in-flight campaign does not reconstruct all of its state and resume deterministically.

Introduce a durable, event-driven coordinator contract: lifecycle-message persistence should wake the owning coordinator, and a restarted coordinator should reconstruct its complete campaign state from the orchestration database without manual re-dispatch.

User-visible failures this would unify

• Orchestration completion can remain unseen without an explicit inbox poll #8698: successful completion remains unseen without an explicit inbox poll.
• [Bug]: A failed dispatch never propagates to the task — dead agents leave tasks 'ready' forever and work is silently dropped #8984: failed dispatches do not propagate to task state, silently dropping work.
• [Bug]: worker_done with null taskId/dispatchId silently fails to complete the task #7429: malformed/null lifecycle provenance does not complete the active task.
• [Bug]: Remote disconnect is misclassified as agent completion #9151: disconnect can be misclassified as completion.
• [Bug]: Terminal/session handle instability breaks orchestration message addressing and injection #9163: routing through changing terminal handles delays or prevents message injection.
• orca status can report ready while individual orchestration RPCs transiently return runtime_unavailable, leaving the coordinator unable to determine whether it should wait, retry, or fail work.

These have separate immediate causes, but they violate the same invariant: every terminal lifecycle outcome must deterministically advance the durable task graph and wake its owner.

Source evidence on current main

• src/main/runtime/orchestration/coordinator.ts runs tick() followed by sleep(pollIntervalMs); messages are read on the next tick rather than waking the loop when persisted.
• The coordinator's completed tasks, failed tasks, escalations, and stop state are held in process memory.
• runFromExistingRun() can reopen a coordinator run row, but restart convergence does not reconstruct the complete prior in-memory state, active wait position, and escalation context.
• lifecycle-reconciliation.ts reconciles lifecycle messages, but it is not a durable process/session supervisor and does not turn every endpoint exit into an authoritative task transition.

Expected invariants

1. Persisting worker_done, escalation, decision_gate, heartbeat, or endpoint-exit state wakes the owning coordinator immediately.
2. Every dispatched attempt reaches one durable terminal outcome: completed, failed, retryable, blocked, or explicitly cancelled.
3. A coordinator restart reconstructs completed/failed sets, active gates, unresolved escalations, attempts, and dependencies from durable state.
4. Delivery and lifecycle authority are keyed by stable dispatch/pane identity rather than a transient terminal handle.
5. Duplicate/replayed messages are idempotent.
6. Aggregate runtime health reflects whether the orchestration subsystem can service a trivial request for the current generation.

Suggested acceptance tests

• Persist worker_done while the coordinator is sleeping; assert it wakes and advances the dependent task without an explicit inbox poll.
• Kill a dispatched worker; assert the dispatch and task converge on an explicit failed/retry state and an escalation is emitted.
• Restart Orca with a three-task DAG containing one completed task, one active decision gate, and one dispatched worker; assert the coordinator resumes exactly once with the same dependency and escalation state.
• Remint a terminal handle while preserving pane/dispatch identity; assert lifecycle messages still route to the correct coordinator.
• Replay the same completion message after restart; assert no duplicate completion or worker launch.
• Make orchestration RPC health fail while the runtime socket remains reachable; assert orca status reports a degraded orchestration subsystem rather than aggregate ready.

Related implementation work

• fix: harden orchestration lifecycle messages #8388, fix(orchestration): fail direct dispatches on agent exit #9001, fix(orchestration): complete null-id worker_done from active dispatch #7790 and fix(orchestration): emit message timestamps as RFC3339 UTC (partial, refs #8698) #9167 harden individual lifecycle paths.
• PR Recover stale coordinator runs #6981 approaches stale coordinator-run recovery.
• PR feat(orchestration): Orcastrator — a persistent director for multi-worktree orchestration #6293 explores a persistent director.

This issue is intended as the durable/event-driven contract tying those paths together, not a replacement for their focused fixes. Prepared with AI assistance from source inspection and live orchestration failures. Reporter: @ericjung.