Runtime status WebSocket events reach only the clients connected to the publishing node

[hasithaa]

Summary

runtimeBroadcaster is an in-process singleton, so a node publishes runtime-status events only
to the WebSocket clients it is holding. With two ICP nodes behind a round-robin proxy, two
browsers land on two nodes and each hears only the events its own node published — roughly half
of them, and silently: the page looks normal and simply never updates.

Where

• icp_server/runtime_ws_service.bal — subscribes each client to the in-process
  storage:runtimeBroadcaster.
• icp_server/modules/storage/heartbeat_repository.bal — publishes on a status change from the
  node that processed that heartbeat.

Observed

Two signed-in consoles (separate browser contexts, so each got its own connection and the proxy
balanced them independently), then one runtime stopped:

client-A: 0 frame(s), 0 about runtime status
client-B: 1 frame(s), 1 about runtime status
          {"eventType":"RUNTIME_STATUS", "environmentId":"750e8400-…", "environmentName":"dev", "runtime…

One of two clients was told. The other was never told and displayed nothing to indicate it had
missed anything.

Steps to reproduce

1. Two ICP nodes on one database, behind a round-robin proxy with no session affinity.
2. Open the console twice (separate browser profiles/contexts) and sign in on both, so each holds
   its own /runtime-status socket. The edge access log confirms the two upgrades went to
   different upstreams.
3. Stop one runtime and wait for it to be marked OFFLINE.
4. Only the console whose socket is on the publishing node receives a RUNTIME_STATUS frame.

Impact

Any console session has a roughly 1/N chance of hearing a given event with N nodes. It is
silent, so it reads as "the runtime list is stale" rather than as a missed event, and there is
no client-side symptom to alert anyone.

There is a second, related gap: markOfflineRuntimes performs a table-wide transition but does
not publish what it transitioned, so even a client on the right node may not learn about
runtimes that went offline via the sweep rather than via a heartbeat.

Suggested direction

• Fan out across nodes rather than in-process — PostgreSQL LISTEN/NOTIFY, a small events
  table the nodes poll, or an external bus, depending on what the deployment already assumes.
• Have markOfflineRuntimes publish the rows it actually transitioned, so sweep-driven
  transitions are announced too.
• Until then, a client-side periodic refetch would at least bound how stale a console can get.

Environment

ICP 2.0.0-SNAPSHOT, PostgreSQL 16, 2 nodes behind an nginx HTTP proxy balancing per request (no
affinity), 4 runtimes. Not specific to any one feature area — found while testing the two-node
environment for #834, and it affects the runtime views generally.