CI has needed reruns almost daily this week. All eight failed attempts from Aug 19 to 21 have the same shape: a browser event (download, dialog, navigation, request/response) reaches the client late or never. It hits all three clients and both engines.
Incidents
- 32514597037 chrome: py async
capture.download returns download with properties, hung until the 600s watchdog
- 32410039192 #1 chrome: Java
CaptureTest.capturesNavigation(), assertion at CaptureTest.java:68
- 32410039192 #2 chrome: py sync
on_download fires on download, watchdog kill
- 32417520934 firefox: py
test_remove_request_listeners, event delivered after removal
- 32311447425 chrome: JS
capture.dialog() returns dialog info, 10s capture timeout
- 32299525981 chrome + firefox: py sync
onDownload hung in both jobs; py async on_download passed after 7m56s
- 32304950839 firefox: py
test_remove_response_listeners, event delivered after removal
- 32519212978 firefox, fork: py
test_remove_response_listeners, event delivered after removal
Analysis
The firefox job in 32299525981 is the clearest case: the download event was not lost, it arrived 7m56s late and the test passed. The remove-listeners failures are the same thing from the other side, an event already in flight lands after the listener is gone.
The Go router subscribes to all events at connect and forwards unconditionally (clicker/internal/api/router.go, routeBrowserToClient), so the delay is either the forwarding path stalling or the browser emitting late under CPU starvation. Each runner handles about 3 suites x 3 browsers on 4 vCPUs, and each tutorial test spawns its own vibium and browser, so starvation is plausible.
Two separate mechanisms:
- Late or missed events. Timeouts fire and the test fails fast, or an assertion sees a post-removal delivery. The remove-listeners tests may partly be a test-design problem: a local unsubscribe cannot recall an event already in the pipe.
- Python hangs that outlive every timeout. Every Python wait has one (captures 10s, downloads 300s, commands 60s), yet three runs hung 8+ minutes with an idle browser and the client waiting on nothing. Prime suspect:
BiDiClient.send() awaits pending setup registrations with no timeout (clients/python/src/vibium/client.py:122-125). One lost setup response blocks every later command in that session.
Next steps
- Add a timeout to the
_pending_setups wait in the Python client. Turns silent watchdog kills into fast failures that name the stuck command.
- Add forwarding timestamps or a delay log to the router so the next incident shows where the latency is.
- Reproduce under CPU load to confirm mechanism 1, then fix or re-spec the remove-listeners tests.
CI has needed reruns almost daily this week. All eight failed attempts from Aug 19 to 21 have the same shape: a browser event (download, dialog, navigation, request/response) reaches the client late or never. It hits all three clients and both engines.
Incidents
capture.download returns download with properties, hung until the 600s watchdogCaptureTest.capturesNavigation(), assertion atCaptureTest.java:68on_download fires on download, watchdog killtest_remove_request_listeners, event delivered after removalcapture.dialog() returns dialog info, 10s capture timeoutonDownloadhung in both jobs; py asyncon_downloadpassed after 7m56stest_remove_response_listeners, event delivered after removaltest_remove_response_listeners, event delivered after removalAnalysis
The firefox job in 32299525981 is the clearest case: the download event was not lost, it arrived 7m56s late and the test passed. The remove-listeners failures are the same thing from the other side, an event already in flight lands after the listener is gone.
The Go router subscribes to all events at connect and forwards unconditionally (
clicker/internal/api/router.go,routeBrowserToClient), so the delay is either the forwarding path stalling or the browser emitting late under CPU starvation. Each runner handles about 3 suites x 3 browsers on 4 vCPUs, and each tutorial test spawns its own vibium and browser, so starvation is plausible.Two separate mechanisms:
BiDiClient.send()awaits pending setup registrations with no timeout (clients/python/src/vibium/client.py:122-125). One lost setup response blocks every later command in that session.Next steps
_pending_setupswait in the Python client. Turns silent watchdog kills into fast failures that name the stuck command.