Repository navigation
feat(web)!: WebDriver BiDi driver replaces Playwright - #1503
okwasniewski wants to merge 284 commits into
Conversation
🟢 e2e web: 93 passedAll 93 tests in 42 files
Tested by e2e 0.23.1 · 1m 7s · web · run artifacts |
🟢 e2e web agent: 34 passed35 agent steps · 26 replayed from cache · 26 model calls · 127.3k tokens (93% cached) · $0.0053 All 34 tests in 5 files
Tested by e2e 0.23.1 · 1m 45s · web · run artifacts |
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 43 model calls · 266.0k tokens (82% cached) · $0.0179 All 20 tests in 1 file
Tested by e2e 0.23.1 · 6m 14s · android-emulator · run artifacts |
🟢 e2e android: 58 passed, 1 skippedAll 59 tests in 7 files
Tested by e2e 0.23.1 · 9m 19s · android-emulator · run artifacts |
🟢 e2e ios: 56 passed, 1 skippedAll 57 tests in 7 files
Tested by e2e 0.23.1 · 6m 49s · ios-simulator · run artifacts |
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 12 replayed from cache · 54 model calls · 322.5k tokens (82% cached) · $0.0217 All 22 tests in 1 file
Tested by e2e 0.23.1 · 7m 57s · ios-simulator · run artifacts |
741aeec to
30f1ce3
Compare
2411751 to
7e2fbc6
Compare
b9422fc to
22d080f
Compare
7e2fbc6 to
53e9af7
Compare
ff9683d to
34d49bb
Compare
The test of a display: contents target with a position of its own asked the browser's hit test at the centre of the button's own bounding box, which is all zeros for an element with no box, so the probe answered false whether or not the text showed and the line could not fail. The probe now asks at the centre of the button's contents (a Range over them) when the button lays out no box, and the test first opens the same page with overflow: visible on the outer box and expects the probe to answer true there, so a false on the overflow: hidden page says the browser clips the text. With overflow: visible on the second page too, the line fails on Chromium and Firefox (expected true to be false); the old probe passed it.
The binding listens to network.beforeRequestSent only once something needs it: a route, headers, credentials, or a networkidle wait. In "goes idle after every 3xx" the first such wait was the last line, so on Chromium the 3xx fetches before it were never counted and the idle wait had nothing of theirs to wait for. Firefox listens from the start, to hold auth challenges. The test now opens the page with waitUntil: 'networkidle' before the fetches, so both browsers count them. Proof it can fail, with temporary edits of settle in network-binding.ts: when a hop the old mayRedirect rule kept (any 3xx with a Location but a 304) also keeps counting, the test times out on Chromium and Firefox; before this change it passed on Chromium under the same edit. The old mayRedirect rule on its own fails it on neither browser, before or after: settle stops counting a hop before it asks whether the hop redirects, so the idle count never depended on that rule. The unit test network-last-hop is what fails on it (300, 305, 306, 399).
isInClosedDetails walked parentElement and closest('details'), which
never cross a shadow boundary or follow a slot. Content a closed
<details> folds away keeps a box on Chromium and Firefox, so such content
read hidden: false and toBeVisible() passed on a folded button:
- a light child slotted into a <details> of its host's shadow root
- anything in the shadow root of a host that sits under a <details>
It now walks the flat tree, as isSkippedContent does: content is folded
when a closed <details> is on the way up and the node on that path under
it is not its first summary. The walk is memoized like its sibling.
Tests (visibility.test.ts): both shapes, open and closed shadow roots, a
plain button and a display: contents one, each with the <details> closed
and open, read three ways: states.hidden, a role query, an observation.
A button slotted into the summary of a closed <details> stays shown.
Before the fix all eight cases failed on Chromium: hidden read false, a
slotted button was listed in the observation, and a display: contents
button matched the role query. On Firefox the four plain-button cases
failed on hidden; the display: contents ones passed there already, since
Firefox lays out no box for folded text. The summary cases passed before
and after.
clippers counted an ancestor as clipping only when its computed overflow was not visible. CSS Containment 2 has paint containment (contain: paint, strict, content) clip a box's contents on both axes whatever overflow computes to, and content-visibility: auto or hidden turn layout and paint containment on. So a link partly cut by a contain: paint box was aimed at the centre of its whole box, the hit test missed, and the tap ended "intercepts pointer events", where the same markup with overflow: hidden taps. holdsPositioned did not count content-visibility as making a containing block either, and counted contain on boxes that take none. One rule each, in dom.ts: - contains(style, kind): layout or paint containment is on when contain names it (or strict, content) or content-visibility is auto or hidden, and never on a box that takes none: a non-atomic inline box, an internal table box but a cell, an internal ruby box, display: contents. holdsPositioned and clippers both read it. - clippers: the used overflow is the computed one, visible where the viewport takes it, and clip in place of visible under paint containment. A root with paint containment clips too, though the viewport takes its overflow. - clipEdge: where a box clips. Chromium and Firefox both clip at the padding edge, and for overflow: clip and paint containment at the overflow clip edge, which overflow-clip-margin moves (its box keyword and its length); overflow: hidden ignores that margin. Clippers were read by their border box, so a target partly under the border of an overflow: hidden box was aimed at the border. The rounding of a border-radius is left to the hit test. - shown: content a content-visibility: hidden ancestor skips is not visible, as the reader's hidden state already has it. A tap on it now ends "element is not visible"; it ended "intercepts pointer events". scrollIntoViewIfNeeded still compares against a clipper's border box, which the browser reports and a flush box equals exactly; the clip edge is a sum of floats. Tests (clip-containers.test.ts, both browsers): a tap lands in the part that shows under contain: paint, strict, content and content-visibility: auto; a target clipped away by each is refused as outside the viewport; content-visibility: auto as a containing block of a fixed target; boxes that take no containment; a root with paint containment; the padding edge past a border for contain: paint, overflow: hidden and overflow: clip; overflow-clip-margin by length and by box; content-visibility: hidden (tap not visible, states.hidden true). 23 of the new cases failed before the fix on Chromium and on Firefox. Six pass before and after and guard the new rule: an inline box and a table row with contain: paint, an inline content-visibility: auto box under a fixed target, a target in the clip margin of a contain: paint box, overflow: hidden with a clip margin, and a target past the placeholder size of a content-visibility: auto box below the fold.
scrollIntoView on a node in a frame scrolled each frame to the centre by the frame's own centre, then asked the node's document whether the node was out of its viewport. For a frame taller than the top viewport, or than a scroller around it, that viewport is the whole frame, so the node read as in view while the page showed another part of the frame: a button at the top of a 3000px frame ended 1140px above a 720px viewport. A tap recovers through its scroll alignments. The scrollIntoView action did not, nor an element screenshot, which only scrolls. After the frames and the node's own document, the action now scrolls the document of each frame holding the node for the node's box, innermost first (scrollThroughFrames in actions.ts). The box is carried through the frame's map into that document's viewport (projectBox in projection.ts), and scrollBoxIntoView scrolls for it as scrollIntoViewIfNeeded does for an element: to the nearest edge of an ancestor that clips the frame, then by center-if-needed in the viewport. No element has that box, so the scrollers are moved by a distance and the box follows the frame's own movement. scrollIntoViewIfNeeded answers the box it ends with, and the alignment rule and the whole-pixel landing that both scrolls use live once in dom.ts. A frame that fits its view is left where the old steps put it: the box is in view by then, so nothing more scrolls. Tests (frame-scroll.test.ts, both browsers), each asserting the button's box in top-viewport coordinates lies inside the viewport: a button at the top, a third of the way down, and at the bottom of a frame taller than the viewport; a tall frame whose centre already shows; a frame nested in a tall frame; a tall frame in a tall frame; a frame taller than its scroller; a tall frame from another origin. Those eight failed before the fix on Chromium and on Firefox. A frame smaller than the viewport ends centred with its button in view, before and after. A unit test covers projectBox.
network.test.ts builds its AttemptNetwork, request and response events, and routes from tests/helpers/network-fake.ts. The fake also names CDP sessions and can number its collectors, for the files that follow.
network-bodies, network-cache, and network-last-hop each built their own AttemptNetwork, events, and route. They bind and open tabs as the tests of network.test.ts do.
The 1268-line file becomes six: network.test.ts keeps intercepts, routes, and the idle window, and basic auth, response waits, a route's fetch, redirect hops, and what a request leaves behind each get a file. Test bodies and title paths are unchanged.
session.test.ts builds its sessions from tests/helpers/session-fake.ts: the unopened session, the peer scripted to open tabs, the opened session, and the one in a browser other clients drive.
session-adopt, attempt-log, and tab-focus each built a BidiSession and scripted the same tabs. Each keeps only what it scripts on top: CDP session names, focus answers, subscription ids, the adopted tab.
BidiInput.send, a fill, and a press on a node move to session-input.test.ts. session.test.ts keeps refs, tabs, and close. Test bodies and title paths are unchanged.
context-tree and frames held the last two copies of the BidiSession literal. Each keeps the peer it scripts.
Moli 1.1.25 on the BiDi driver: pages open and read, 18 of 79 testbed tests pass. Its CDP returns an element from deep serialization with no node id, so no action reaches a located node, and it has no document.getAnimations, so a screenshot throws.
070f52c to
0f4e261
Compare
As a unit test it loaded @e2e-dev/core through Vite's runner, and Vitest kept that choice for the integration project: core.test.ts then got launch from the runner and createAgent from outside it, each with its own session registry, and createAgent refused every session. Only a run of both projects in one process showed it.
The toHaveRole test from main pinned the old driver's split: the tree read a contenteditable heading as a textbox while getByRole found it as a heading. This driver queries the roles the tree reads. The expect reference, the migration page, and the changeset say so.
Neither status can carry content, and Firefox answers network.getData for a 304 two to four seconds late once the session has run a few attempts (measured over raw BiDi too: 1 ms at first, then seconds). body() of a waited 304 took that long on Firefox and 0 ms on Chromium.
Firefox's own text for console.log(element) is the word node, though it serializes the tag and attributes. The line now reads <output#total.sum>, as Chromium's does.
body() of a response over 20 MB failed with the browser's raw 'no such network data: Collector ... didn't collect response data'.
Measured on Firefox 159: route.fetch() replays a binary request body of 14.99 MiB and refuses 15.01 MiB, text up to 20 MiB; a logged event reads object and a class instance loses its name.
A page starts a dedicated worker that POSTs 1 KB, 64 KB and 1 MB of text and of binary bytes while a route is registered: one that lets the request go on, one that replays it with route.fetch(), and one that matches another URL. The server must receive the exact bytes and the worker the response. A GET, and a 307 of a 64 KB POST whose second hop must carry the route's headers, are covered too. Firefox passes. On Chromium every POST of 64 KB or more never settles, and one of 1 KB now and then: 13 of 20 cases fail per run, in every run. The next commit fixes the cause.
With a route registered, a fetch from a dedicated worker could hang for good on Chromium: every POST of 64 KB or more, one of 1 KB now and then. A dedicated worker has no Fetch domain. Chrome pauses its requests on the page's CDP session and announces them (Network.requestWillBeSent) on the worker's, in either order. The chromium-bidi mapper keeps a request on the target whose session sent its latest event and asks that target's subscriptions and intercepts whether the request is held; a worker's target is under none that names a user context or a tab. A request paused first is on the worker's target once the mapper has both events, so the mapper expects no intercept, waits for the request's ExtraInfo, which Chrome sends only once the request is continued, and neither reports the request nor continues it. A CDP dump shows the fixups hand both events on at once, the mapper log "Request 64427.3 was moved from <page target> to <worker target>", and nothing after it. The fixups kept a request per session, so they could not tell that the pause on one session and the announcement on another were one request. A request is now kept once, by its id, with the sessions that report it, and dedicated workers' sessions are known from Target.attachedToTarget. A pause the mapper has yet to answer when a worker's session first announces its request is handed to the mapper again after the announcement: the request is back on the page's target with both events, and the mapper reports it blocked. The trigger is the pair of events itself, no timer, size, or retry. The same model fixes what a worker's redirect hit once its first hop went through: the hold of an early redirect pause counted announcements on the pause's own session only, so a 307 of a 64 KB POST was continued without the route's headers and reported as the response of its second hop (5 of 5). The pause now waits for the hop the worker's session announces, and the next hop's response ExtraInfo for that redirect. A session that detaches takes what it paused and the requests no other session reported; pauses another session holds for a hop it was to announce go to the mapper instead of waiting for good. tests/integration/route-worker.test.ts on Chromium: 13 of 20 cases failed in 3 of 3 runs before, 20 of 20 pass after.
Plain connect to an Electron app failed with the raw 'browser.createUserContext: unknown error: Failed to create browser context.' as ENGINE_FAILURE. It is UNSUPPORTED_CAPABILITY now, with the browser's words and the option that drives such a host. Run on Electron 45.3.3: plain connect fails with it, context 'existing' passes.
|
[claude-fable-5-2] responding on behalf of okwasniewski @kdy1 thanks for both reviews. The six inline threads have their own replies. The findings that had no thread, by review: First review
Second review, the two smaller items
Found while rerunning your repros, also fixed
Left for Oskar to answer
|
Node 22.22.3 ships undici 6.24.1, whose WebSocket fails the driver twice: - It always offers permessage-deflate, Chrome takes it, and a message that inflates past 4 MB closes the connection (4.0 MB passes, 4.3 MB does not). A 6 MB response body, a 6 MB value from a page call, or an 800 KB binary POST under a route ended the attempt with 'the chromium-bidi mapper exited (code 0)'. - Its mask buffer is 16386 bytes, so every frame after a process's 4096th is masked with zeros. Firefox closes a connection that sends one, then every new one: a probe got 4096 answers and lost the socket at command 4098, and check-actions.test.ts failed 3 of 3 runs. Both browser sockets now use ws with compression off (browser-socket.ts). The mapper gets each CDP message in a task of its own, as it did from the old client's inflate step: handed a network read's messages in a row, it failed the attach tests 3 of 3. On Node 22.22.3 and Node 24: the whole web suite passes on both browsers.
Summary
@e2e-dev/webdrives Chromium and Firefox over WebDriver BiDi. Playwright is removed: noplaywright-coredependency, no WebKit, nosurfaceOf.WebSocket. Firefox speaks BiDi natively; Chromium runs thechromium-bidimapper in a child process. No chromedriver or geckodriver.npx @e2e-dev/web install, each checked against a pinned SHA-256.credentialOrigins(fix(web)!: scope headers and basicAuth to the app's origin #1393), CDP handshake headers (feat(browser-run): add Cloudflare hosted browsers #1634),readStyle(feat(agent): read computed styles in project tools #1633), tabs (feat(web): tabs in the browser fixture, with switch_tab and close_tab agent verbs #1710), corelaunch(refactor: move e2e's core into @e2e-dev/core, and engines peer on it #1737), check through a covering label (fix(web): list a checkbox or radio its own label draws over no box #1595), screenshot retry after navigation (fix: page a list that appends at the end, and retry a screenshot after navigation #1614).dragToon an HTML5draggabledropped nothing silently (nowUNSUPPORTED_CAPABILITY, bug 1515879).web (windows)CI job: launch, keyboard, and tabs on both browsers.What breaks is in
.changeset/web-bidi-only.md. Limits are indocs/reference/web.mdx.Benchmark totals (before the rebase; 20 warm / 10 cold iterations, M5 Pro, headless)
Firefox cold launch is slower: the Remote Agent holds
session.newuntil startup finishes (925 ms vs 314 ms), once per worker.Issue
Spike, no issue.
Verified
Ran it locally: yes
@e2e-dev/web: 1483 tests on Chromium and Firefox, 0 failures. One env-gated skip: the crosscheck benchmark scenarios, which run inbenchmark.yml.e2e4718 and core tests pass. Build, typecheck, oxlint, fallow, docs, plugin, MCP, peer-range, and changeset checks are clean.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it.docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.