Skip to content

Browser events reach client listeners late or not at all #397

Description

@vincebln2

CI has needed reruns almost daily this week. All eight failed attempts from Aug 19 to 21 have the same shape: a browser event (download, dialog, navigation, request/response) reaches the client late or never. It hits all three clients and both engines.

Incidents

  • 32514597037 chrome: py async capture.download returns download with properties, hung until the 600s watchdog
  • 32410039192 #1 chrome: Java CaptureTest.capturesNavigation(), assertion at CaptureTest.java:68
  • 32410039192 #2 chrome: py sync on_download fires on download, watchdog kill
  • 32417520934 firefox: py test_remove_request_listeners, event delivered after removal
  • 32311447425 chrome: JS capture.dialog() returns dialog info, 10s capture timeout
  • 32299525981 chrome + firefox: py sync onDownload hung in both jobs; py async on_download passed after 7m56s
  • 32304950839 firefox: py test_remove_response_listeners, event delivered after removal
  • 32519212978 firefox, fork: py test_remove_response_listeners, event delivered after removal

Analysis

The firefox job in 32299525981 is the clearest case: the download event was not lost, it arrived 7m56s late and the test passed. The remove-listeners failures are the same thing from the other side, an event already in flight lands after the listener is gone.

The Go router subscribes to all events at connect and forwards unconditionally (clicker/internal/api/router.go, routeBrowserToClient), so the delay is either the forwarding path stalling or the browser emitting late under CPU starvation. Each runner handles about 3 suites x 3 browsers on 4 vCPUs, and each tutorial test spawns its own vibium and browser, so starvation is plausible.

Two separate mechanisms:

  1. Late or missed events. Timeouts fire and the test fails fast, or an assertion sees a post-removal delivery. The remove-listeners tests may partly be a test-design problem: a local unsubscribe cannot recall an event already in the pipe.
  2. Python hangs that outlive every timeout. Every Python wait has one (captures 10s, downloads 300s, commands 60s), yet three runs hung 8+ minutes with an idle browser and the client waiting on nothing. Prime suspect: BiDiClient.send() awaits pending setup registrations with no timeout (clients/python/src/vibium/client.py:122-125). One lost setup response blocks every later command in that session.

Next steps

  1. Add a timeout to the _pending_setups wait in the Python client. Turns silent watchdog kills into fast failures that name the stuck command.
  2. Add forwarding timestamps or a delay log to the router so the next incident shows where the latency is.
  3. Reproduce under CPU load to confirm mechanism 1, then fix or re-spec the remove-listeners tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions