Skip to content

feat(web)!: WebDriver BiDi driver replaces Playwright - #1503

Open
okwasniewski wants to merge 284 commits into
mainfrom
oskar/webdriver-bidi
Open

okwasniewski wants to merge 284 commits into
mainfrom
oskar/webdriver-bidi

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Summary

What breaks is in .changeset/web-bidi-only.md. Limits are in docs/reference/web.mdx.

Benchmark totals (before the rebase; 20 warm / 10 cold iterations, M5 Pro, headless)

Total PW chromium BiDi chromium PW firefox BiDi firefox
sum of p50s, every scenario (ms) 1534 981 (-36%) 2623 2116 (-19%)
cold launch + new page + 6 agent steps (ms) 334 274 (-18%) 768 1279 (+66%)
testbed wall clock, median of 3 (ms) 17525 16895 (-4%) 21102 20980 (-1%)
node + browser CPU, warm loop (s) 51.7 20.3 (-61%) 77.7 33.7 (-57%)
peak RSS, node + browser (MB) 1405 1542 (+10%) 2351 2644 (+12%)

Firefox cold launch is slower: the Remote Agent holds session.new until startup finishes (925 ms vs 314 ms), once per worker.

Issue

Spike, no issue.

Verified

Ran it locally: yes

  • @e2e-dev/web: 1483 tests on Chromium and Firefox, 0 failures. One env-gated skip: the crosscheck benchmark scenarios, which run in benchmark.yml.
  • Testbed through the built CLI on Chromium, and on Firefox without the Playwright-tagged tests: all pass.
  • e2e 4718 and core tests pass. Build, typecheck, oxlint, fallow, docs, plugin, MCP, peer-range, and changeset checks are clean.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it.
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

@github-actions github-actions Bot added the size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 93 passed

All 93 tests in 42 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 609ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 433ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 176ms
🟢 tests/async-states.e2e.ts · 2 passed 6.9s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.7s
🟢 tests/canvas-only.e2e.ts · 2 passed 460ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 246ms
🟢 canvas only › a shape out of order paints the error and resets progress 214ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 602ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 376ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 226ms
🟢 tests/cart-totals.e2e.ts · 2 passed 355ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 195ms
🟢 cart totals › a quantity never drops below one 160ms
🟢 tests/checkout-review.e2e.ts · 1 passed 136ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 136ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 510ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 283ms
🟢 context menu trap › the decoy menu items refuse and keep the file 227ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.0s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 230ms
🟢 control inventory › check picks a radio the pick replaces with its summary 145ms
🟢 control inventory › a long press and a double click are told apart from a tap 926ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 204ms
🟢 control inventory › setInputFiles attaches files named from the project root 116ms
🟢 control inventory › the details view is left through the browser history 144ms
🟢 control inventory › scrollIntoView brings the footnote into view 134ms
🟢 control inventory › every exercise done shows the success message 1.2s
🟢 tests/css-content-ui.e2e.ts · 2 passed 320ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 165ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 155ms
🟢 tests/date-picker.e2e.ts · 2 passed 520ms
🟢 date picker › navigates three months ahead and books the required date 301ms
🟢 date picker › confirming without the exact date shows the error 219ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.7s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.3s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.0s
🟢 one submit opens one confirm, even when double-clicked 1.0s
🟢 tests/div-soup.e2e.ts · 2 passed 577ms
🟢 div soup › a wrong code is rejected 261ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 316ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 497ms
🟢 drag and drop › an out-of-order item resets the dropzone 250ms
🟢 drag and drop › dropping Banana then Cherry passes 247ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 177ms
🟢 filter deep link › filtering writes the query to the URL 91ms
🟢 filter deep link › opening a shared link restores the filter 86ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 492ms
🟢 gift card purchase › refuses to buy without an amount 138ms
🟢 gift card purchase › refuses a recipient without an at sign 165ms
🟢 gift card purchase › sends the gift card 189ms
🟢 tests/hover-menu.e2e.ts · 2 passed 389ms
🟢 hover menu › a decoy leaf is refused 159ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 230ms
🟢 tests/iframe-form.e2e.ts · 2 passed 327ms
🟢 iframe form › a wrong code is refused by the host page 159ms
🟢 iframe form › the code read from one frame applies through the other 168ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 352ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 89ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 117ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 146ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.7s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.7s
🟢 tests/kanban-board.e2e.ts · 3 passed 767ms
🟢 kanban board › pointer drags move both cards into the goal layout 415ms
🟢 kanban board › a swipe along a path drags a card as well 181ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 171ms
🟢 tests/login-form.e2e.ts · 4 passed 784ms
🟢 login form › rejects a malformed email 186ms
🟢 login form › rejects a short password 144ms
🟢 login form › rejects an unknown account 168ms
🟢 login form › signs in and logs out 286ms
🟢 tests/lying-labels.e2e.ts · 4 passed 880ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 212ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 147ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 256ms
🟢 lying labels › visible text drives the payment through 265ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 518ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 221ms
🟢 native dialogs › dismissing the confirm cancels the order 163ms
🟢 native dialogs › a wrong gift code is rejected 134ms
🟢 tests/new-tab-flow.e2e.ts · 2 passed 330ms
🟢 new tab flow › reads the code in the tab the app opens and enters it back in the original tab 160ms
🟢 new tab flow › a tap that opens a tab leaves the screen on the original tab, and activate shows the new one 170ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.1s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.1s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.3s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 281ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 314ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.2s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.4s
🟢 tests/order-history.e2e.ts · 2 passed 1.3s
🟢 order history › overview and settings render their panels 162ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.2s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.3s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 422ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 100ms
🟢 otp auto-advance › a wrong code clears every box and starts over 740ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 697ms
🟢 continues only after the banner and the overlay are gone 697ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 496ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 254ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 242ms
🟢 tests/price-sorting.e2e.ts · 3 passed 361ms
🟢 price sorting › opens in the featured order with sale prices struck through 115ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 120ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 126ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 440ms
🟢 rich text editor › bolding exactly the required word passes the check 224ms
🟢 rich text editor › bold that spills into the previous word fails the check 216ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 473ms
🟢 saved addresses › adds an address and clears the form 149ms
🟢 saved addresses › ignores an add with an empty field 132ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 192ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 324ms
🟢 shadow DOM form › rejects a wrong access code 160ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 164ms
🟢 tests/smoke.e2e.ts · 1 passed 247ms
🟢 home lists the scenarios and opens one 247ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.1s
🟢 taps the moving target three times across rebuilds of the list 2.1s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 325ms
🟢 accepts the terms once the button clears the sticky footer 325ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 21.7s
🟢 scrolls the windowed table to the Golden Row and claims it 21.7s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 242ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.2s

Tested by e2e 0.23.1 · 1m 7s · web · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 26 model calls · 127.3k tokens (93% cached) · $0.0053

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.6s
🟢 control inventory › check sets the checkbox and the radio 1 step 681ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 297ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 904ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.7s
🟢 control inventory › upload attaches the named project files 1 step 2.5s
🟢 control inventory › back returns from the details view to the inventory 1 step 258ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 277ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 347ms
🟢 act signs in with the benchmark account 1 step 347ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 12.4s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 6.3s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.5s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 4.5s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 18 calls 1m 24s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.3s
🟢 act completes lying-labels 1 step · 4 calls 6.5s
🟢 act completes iframe-form 1 step · 3 calls 4.2s
🟢 act completes infinite-scroll 1 step 3.2s
🟢 act completes virtualized-table 1 step 22.5s
🟢 act completes overlay-trap 1 step 759ms
🟢 act completes sticky-chrome 1 step 956ms
🟢 act completes async-states 1 step · 3 calls 7.4s
🟢 act completes debounced-search 1 step 2.2s
🟢 act completes hover-menu 1 step 366ms
🟢 act completes drag-and-drop 1 step 368ms
🟢 act completes kanban-board 1 step 566ms
🟢 act completes context-menu-trap 1 step 2.5s
🟢 act completes stale-dom 1 step 376ms
🟢 act completes otp-auto-advance 1 step 584ms
🟢 act completes date-picker 1 step 684ms
🟢 act completes rich-text-editor 1 step 9.8s
🟢 act completes playbook-cleanup 1 step 451ms
🟢 act completes promo-storefront 1 step 2.2s
🟢 act completes new-tab-flow 1 step · 5 calls 11.6s
🟢 act completes gift-card-purchase 1 step 2.3s
🟢 act filters the list and the URL carries the filter 1 step 161ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 204ms
🟢 act adds an address with a run-unique label 1 step 204ms

Tested by e2e 0.23.1 · 1m 45s · web · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 43 model calls · 266.0k tokens (82% cached) · $0.0179

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 43 calls 6m 13s
🟢 act completes Login Form 1 step 26.5s
🟢 act completes Infinite Scroll List 1 step 21.3s
🟢 act completes Modal Flow 1 step 12.5s
🟢 act completes Bottom Tabs 1 step 8.0s
🟢 act completes Text Input Variations 1 step 7.0s
🟢 act completes Broken Accessibility 2 steps · 5 calls 13.1s
🟢 act completes Vision Only 2 steps · 1 call 6.4s
🟢 act completes Gestures 1 step · 9 calls 23.7s
🟢 act completes Async States 1 step · 15 calls 37.6s
🟢 act completes Debounced Search 1 step 7.1s
🟢 act completes Huge Virtualized List 1 step 1m 51s
🟢 act completes Flattened Registration Form 2 steps · 1 call 9.9s
🟢 act completes Flattened Login 2 steps · 6 calls 15.4s
🟢 act completes Sticky Chrome Target 1 step 10.0s
🟢 act completes WebView Accessibility 1 step · 6 calls 19.3s
🟢 act completes Permission Prompt 1 step 7.4s
🟢 act completes Product Catalog 1 step 8.7s
🟢 act completes Error Recovery 1 step 9.3s
🟢 act completes Choice Controls 1 step 8.8s
🟢 act completes Sequential Onboarding 1 step 10.4s

Tested by e2e 0.23.1 · 6m 14s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 58 passed, 1 skipped

All 59 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 44s
🟢 control inventory › a label query resolves a labeled text field 19.6s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 12.0s
🟢 control inventory › a placeholder query and attribute reads find the field 7.3s
🟢 control inventory › a secure field takes a fill and denies value reads 6.8s
🟢 control inventory › press sends a character and Enter through the soft keyboard 7.7s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 8.2s
🟢 control inventory › toBeFocused follows the focus 8.0s
🟢 control inventory › a switch checks, unchecks, and reports its state 8.3s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.2s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 10.4s
🟢 control inventory › a tab role answers getByRole 3.9s
🟢 control inventory › a radio group reports the checked option 7.5s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 6.6s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.5s
🟢 control inventory › a header answers its accessible name 3.8s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.7s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 5.8s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.2s
🟢 control inventory › a double tap lands inside the 300 ms window 6.4s
🟢 control inventory › an ambiguous locator fails at once 5.0s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.2s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 19s
🟢 device fixture › setOrientation rotates the window the app measures 5.0s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.8s
🟢 device fixture › foregroundApp sees the home screen after home() 4.2s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.3s
🟢 device fixture › closeApp leaves the next commands on the same device 6.4s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.4s
🟢 device fixture › openLink opens the scenario a deep link names 4.4s
🟢 device fixture › device.back pops the scenario 4.3s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 3.6s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.1s
🟢 device fixture › setLocation and clearLocation change what the app reads 20.6s
🟢 tests/device.e2e.ts · 5 passed 39.3s
🟢 permission prompt › allowing the system dialog enables the microphone 5.6s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 11.9s
🟢 sequential onboarding echoes the exact values 9.4s
🟢 onboarding rejects a malformed email before advancing 5.3s
🟢 dismissing the keyboard over choice controls leaves them as they were 7.1s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 14s
🟢 infinite scroll reaches the target item 28.4s
🟢 product catalog adds exactly two of the right variant 8.3s
🟢 gestures: long-press, then swipe left 7.8s
🟢 gestures: double-tap completes the flow 9.6s
🟢 sticky chrome: accept below the fold, then continue 12.8s
🟢 async states: load, pull to refresh, claim through the toast 13.4s
🟢 huge virtualized list reaches row 512 1m 53s
🟢 tests/flows.e2e.ts · 7 passed 48.8s
🟢 text inputs unlock the submit button 5.8s
🟢 filled text inputs report their values 5.5s
🟢 modal flow confirms through the native alert 8.3s
🟢 bottom tabs act inside the last tab 6.2s
🟢 error recovery retries, then confirms the delete 8.3s
🟢 choice controls place the exact order 6.0s
🟢 debounced search selects the target once results arrive 8.6s
🟢 tests/login-form.e2e.ts · 3 passed 27.4s
🟢 login form › rejects a malformed email 7.4s
🟢 login form › rejects an unknown account 7.5s
🟢 login form › signs in and logs out 12.5s
🟢 tests/smoke.e2e.ts · 1 passed 5.6s
🟢 home lists the scenarios and opens one 5.6s

Tested by e2e 0.23.1 · 9m 19s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 56 passed, 1 skipped

All 57 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 2s
🟢 control inventory › a label query resolves a labeled text field 12.5s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.7s
🟢 control inventory › a placeholder query and attribute reads find the field 4.6s
🟢 control inventory › a secure field takes a fill and denies value reads 7.5s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.0s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 9.9s
🟢 control inventory › toBeFocused follows the focus 9.2s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.1s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 10.3s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.5s
🟢 control inventory › a radio group reports the checked option 9.8s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.0s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.6s
🟢 control inventory › a header answers its accessible name 4.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.3s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.6s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.4s
🟢 control inventory › pointer verbs land at the point asked for 9.4s
🟢 control inventory › an ambiguous locator fails at once 6.4s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 12.1s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 168ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 12 passed 1m 53s
🟢 device fixture › setAppearance flips the color scheme the app reads 16.0s
🟢 device fixture › setOrientation rotates the window the app measures 6.6s
🟢 device fixture › the clipboard reads back what was written 5.4s
🟢 device fixture › home leaves the app, openApp brings it back where it was 6.8s
🟢 device fixture › closeApp then openApp relaunches on the home list 9.3s
🟢 device fixture › closeApp leaves the next commands on the same device 13.6s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 5.8s
🟢 device fixture › openLink opens the scenario a deep link names 15.6s
🟢 device fixture › device.back pops the scenario 8.1s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.4s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.2s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 11.6s
🟢 tests/device.e2e.ts · 3 passed 49.3s
🟢 sequential onboarding echoes the exact values 27.9s
🟢 onboarding rejects a malformed email before advancing 10.0s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.3s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 6s
🟢 infinite scroll reaches the target item 37.4s
🟢 product catalog adds exactly two of the right variant 10.3s
🟢 gestures: long-press, then swipe left 8.8s
🟢 sticky chrome: accept below the fold, then continue 12.4s
🟢 async states: load, pull to refresh, claim through the toast 12.5s
🟢 huge virtualized list reaches row 512 2m 32s
🟢 web view coupon form applies the code shown on the page 12.5s
🟢 tests/flows.e2e.ts · 7 passed 1m 10s
🟢 text inputs unlock the submit button 12.5s
🟢 filled text inputs report their values 10.4s
🟢 modal flow confirms through the native alert 9.8s
🟢 bottom tabs act inside the last tab 7.3s
🟢 error recovery retries, then confirms the delete 9.3s
🟢 choice controls place the exact order 8.5s
🟢 debounced search selects the target once results arrive 12.0s
🟢 tests/login-form.e2e.ts · 3 passed 45.0s
🟢 login form › rejects a malformed email 13.0s
🟢 login form › rejects an unknown account 12.8s
🟢 login form › signs in and logs out 19.2s
🟢 tests/smoke.e2e.ts · 1 passed 7.5s
🟢 home lists the scenarios and opens one 7.5s

Tested by e2e 0.23.1 · 6m 49s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 54 model calls · 322.5k tokens (82% cached) · $0.0217

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 54 calls 7m 56s
🟢 act completes Login Form 1 step 37.3s
🟢 act completes Infinite Scroll List 1 step 40.4s
🟢 act completes Modal Flow 1 step 10.6s
🟢 act completes Bottom Tabs 1 step 7.3s
🟢 act completes Text Input Variations 1 step 12.7s
🟢 act completes Broken Accessibility 2 steps · 6 calls 27.0s
🟢 act completes Vision Only 2 steps · 6 calls 28.1s
🟢 act completes Gestures 1 step · 9 calls 40.8s
🟢 act completes Async States 1 step · 8 calls 24.1s
🟢 act completes Debounced Search 1 step · 1 call 26.8s
🟢 act completes Flattened Registration Form 2 steps · 1 call 26.0s
🟢 act completes Flattened Login 2 steps · 11 calls 54.9s
🟢 act completes Sticky Chrome Target 1 step 13.2s
🟢 act completes WebView Accessibility 1 step 12.0s
🟢 act completes Permission Prompt 1 step 11.2s
🟢 act completes Photo Picker 1 step · 2 calls 29.6s
🟢 act completes Product Catalog 1 step 10.0s
🟢 act completes Error Recovery 1 step 10.3s
🟢 act completes Choice Controls 1 step · 5 calls 32.8s
🟢 act completes Stripe PaymentSheet 1 step · 5 calls 10.7s
🟢 act completes Apple Pay 1 step 10.7s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.1 · 7m 57s · ios-simulator · run artifacts

@okwasniewski
okwasniewski changed the base branch from main to oskar/playwright-parity-tests November 1, 2026 18:40
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 2 times, most recently from 2411751 to 7e2fbc6 Compare November 1, 2026 19:12
@okwasniewski
okwasniewski force-pushed the oskar/playwright-parity-tests branch 2 times, most recently from b9422fc to 22d080f Compare November 4, 2026 16:57
Base automatically changed from oskar/playwright-parity-tests to main November 4, 2026 18:02
@okwasniewski okwasniewski changed the title feat(web): experimental WebDriver BiDi driver feat(web)!: WebDriver BiDi driver replaces Playwright Nov 6, 2026
Comment thread packages/web/src/install/download.ts Dismissed
@okwasniewski
okwasniewski marked this pull request as ready for review November 16, 2026 17:20
@okwasniewski
okwasniewski force-pushed the oskar/webdriver-bidi branch 7 times, most recently from ff9683d to 34d49bb Compare November 16, 2026 17:57
The test of a display: contents target with a position of its own asked
the browser's hit test at the centre of the button's own bounding box,
which is all zeros for an element with no box, so the probe answered
false whether or not the text showed and the line could not fail.

The probe now asks at the centre of the button's contents (a Range over
them) when the button lays out no box, and the test first opens the same
page with overflow: visible on the outer box and expects the probe to
answer true there, so a false on the overflow: hidden page says the
browser clips the text.

With overflow: visible on the second page too, the line fails on Chromium
and Firefox (expected true to be false); the old probe passed it.
The binding listens to network.beforeRequestSent only once something
needs it: a route, headers, credentials, or a networkidle wait. In "goes
idle after every 3xx" the first such wait was the last line, so on
Chromium the 3xx fetches before it were never counted and the idle wait
had nothing of theirs to wait for. Firefox listens from the start, to
hold auth challenges.

The test now opens the page with waitUntil: 'networkidle' before the
fetches, so both browsers count them.

Proof it can fail, with temporary edits of settle in network-binding.ts:
when a hop the old mayRedirect rule kept (any 3xx with a Location but a
304) also keeps counting, the test times out on Chromium and Firefox;
before this change it passed on Chromium under the same edit.

The old mayRedirect rule on its own fails it on neither browser, before
or after: settle stops counting a hop before it asks whether the hop
redirects, so the idle count never depended on that rule. The unit test
network-last-hop is what fails on it (300, 305, 306, 399).
isInClosedDetails walked parentElement and closest('details'), which
never cross a shadow boundary or follow a slot. Content a closed
<details> folds away keeps a box on Chromium and Firefox, so such content
read hidden: false and toBeVisible() passed on a folded button:

- a light child slotted into a <details> of its host's shadow root
- anything in the shadow root of a host that sits under a <details>

It now walks the flat tree, as isSkippedContent does: content is folded
when a closed <details> is on the way up and the node on that path under
it is not its first summary. The walk is memoized like its sibling.

Tests (visibility.test.ts): both shapes, open and closed shadow roots, a
plain button and a display: contents one, each with the <details> closed
and open, read three ways: states.hidden, a role query, an observation.
A button slotted into the summary of a closed <details> stays shown.

Before the fix all eight cases failed on Chromium: hidden read false, a
slotted button was listed in the observation, and a display: contents
button matched the role query. On Firefox the four plain-button cases
failed on hidden; the display: contents ones passed there already, since
Firefox lays out no box for folded text. The summary cases passed before
and after.
clippers counted an ancestor as clipping only when its computed overflow
was not visible. CSS Containment 2 has paint containment (contain: paint,
strict, content) clip a box's contents on both axes whatever overflow
computes to, and content-visibility: auto or hidden turn layout and paint
containment on. So a link partly cut by a contain: paint box was aimed at
the centre of its whole box, the hit test missed, and the tap ended
"intercepts pointer events", where the same markup with overflow: hidden
taps. holdsPositioned did not count content-visibility as making a
containing block either, and counted contain on boxes that take none.

One rule each, in dom.ts:

- contains(style, kind): layout or paint containment is on when contain
  names it (or strict, content) or content-visibility is auto or hidden,
  and never on a box that takes none: a non-atomic inline box, an
  internal table box but a cell, an internal ruby box, display: contents.
  holdsPositioned and clippers both read it.
- clippers: the used overflow is the computed one, visible where the
  viewport takes it, and clip in place of visible under paint
  containment. A root with paint containment clips too, though the
  viewport takes its overflow.
- clipEdge: where a box clips. Chromium and Firefox both clip at the
  padding edge, and for overflow: clip and paint containment at the
  overflow clip edge, which overflow-clip-margin moves (its box keyword
  and its length); overflow: hidden ignores that margin. Clippers were
  read by their border box, so a target partly under the border of an
  overflow: hidden box was aimed at the border. The rounding of a
  border-radius is left to the hit test.
- shown: content a content-visibility: hidden ancestor skips is not
  visible, as the reader's hidden state already has it. A tap on it now
  ends "element is not visible"; it ended "intercepts pointer events".

scrollIntoViewIfNeeded still compares against a clipper's border box,
which the browser reports and a flush box equals exactly; the clip edge
is a sum of floats.

Tests (clip-containers.test.ts, both browsers): a tap lands in the part
that shows under contain: paint, strict, content and content-visibility:
auto; a target clipped away by each is refused as outside the viewport;
content-visibility: auto as a containing block of a fixed target; boxes
that take no containment; a root with paint containment; the padding
edge past a border for contain: paint, overflow: hidden and overflow:
clip; overflow-clip-margin by length and by box; content-visibility:
hidden (tap not visible, states.hidden true).

23 of the new cases failed before the fix on Chromium and on Firefox.
Six pass before and after and guard the new rule: an inline box and a
table row with contain: paint, an inline content-visibility: auto box
under a fixed target, a target in the clip margin of a contain: paint
box, overflow: hidden with a clip margin, and a target past the
placeholder size of a content-visibility: auto box below the fold.
scrollIntoView on a node in a frame scrolled each frame to the centre by
the frame's own centre, then asked the node's document whether the node
was out of its viewport. For a frame taller than the top viewport, or
than a scroller around it, that viewport is the whole frame, so the node
read as in view while the page showed another part of the frame: a
button at the top of a 3000px frame ended 1140px above a 720px viewport.
A tap recovers through its scroll alignments. The scrollIntoView action
did not, nor an element screenshot, which only scrolls.

After the frames and the node's own document, the action now scrolls
the document of each frame holding the node for the node's box,
innermost first (scrollThroughFrames in actions.ts). The box is carried
through the frame's map into that document's viewport (projectBox in
projection.ts), and scrollBoxIntoView scrolls for it as
scrollIntoViewIfNeeded does for an element: to the nearest edge of an
ancestor that clips the frame, then by center-if-needed in the
viewport. No element has that box, so the scrollers are moved by a
distance and the box follows the frame's own movement.
scrollIntoViewIfNeeded answers the box it ends with, and the alignment
rule and the whole-pixel landing that both scrolls use live once in
dom.ts.

A frame that fits its view is left where the old steps put it: the box
is in view by then, so nothing more scrolls.

Tests (frame-scroll.test.ts, both browsers), each asserting the
button's box in top-viewport coordinates lies inside the viewport: a
button at the top, a third of the way down, and at the bottom of a
frame taller than the viewport; a tall frame whose centre already
shows; a frame nested in a tall frame; a tall frame in a tall frame; a
frame taller than its scroller; a tall frame from another origin. Those
eight failed before the fix on Chromium and on Firefox. A frame smaller
than the viewport ends centred with its button in view, before and
after. A unit test covers projectBox.
network.test.ts builds its AttemptNetwork, request and response events,
and routes from tests/helpers/network-fake.ts. The fake also names CDP
sessions and can number its collectors, for the files that follow.
network-bodies, network-cache, and network-last-hop each built their
own AttemptNetwork, events, and route. They bind and open tabs as the
tests of network.test.ts do.
The 1268-line file becomes six: network.test.ts keeps intercepts,
routes, and the idle window, and basic auth, response waits, a route's
fetch, redirect hops, and what a request leaves behind each get a file.
Test bodies and title paths are unchanged.
session.test.ts builds its sessions from tests/helpers/session-fake.ts:
the unopened session, the peer scripted to open tabs, the opened
session, and the one in a browser other clients drive.
session-adopt, attempt-log, and tab-focus each built a BidiSession and
scripted the same tabs. Each keeps only what it scripts on top: CDP
session names, focus answers, subscription ids, the adopted tab.
BidiInput.send, a fill, and a press on a node move to
session-input.test.ts. session.test.ts keeps refs, tabs, and close.
Test bodies and title paths are unchanged.
context-tree and frames held the last two copies of the BidiSession
literal. Each keeps the peer it scripts.
Moli 1.1.25 on the BiDi driver: pages open and read, 18 of 79 testbed tests pass. Its CDP returns an element from deep serialization with no node id, so no action reaches a located node, and it has no document.getAnimations, so a screenshot throws.
Comment thread packages/web/src/in-page/dom.ts Dismissed
As a unit test it loaded @e2e-dev/core through Vite's runner, and Vitest kept that choice for the integration project: core.test.ts then got launch from the runner and createAgent from outside it, each with its own session registry, and createAgent refused every session. Only a run of both projects in one process showed it.
The toHaveRole test from main pinned the old driver's split: the tree read a contenteditable heading as a textbox while getByRole found it as a heading. This driver queries the roles the tree reads. The expect reference, the migration page, and the changeset say so.
Neither status can carry content, and Firefox answers network.getData for a 304 two to four seconds late once the session has run a few attempts (measured over raw BiDi too: 1 ms at first, then seconds). body() of a waited 304 took that long on Firefox and 0 ms on Chromium.
Firefox's own text for console.log(element) is the word node, though it serializes the tag and attributes. The line now reads <output#total.sum>, as Chromium's does.
body() of a response over 20 MB failed with the browser's raw 'no such network data: Collector ... didn't collect response data'.
Measured on Firefox 159: route.fetch() replays a binary request body of 14.99 MiB and refuses 15.01 MiB, text up to 20 MiB; a logged event reads object and a class instance loses its name.
A page starts a dedicated worker that POSTs 1 KB, 64 KB and 1 MB of
text and of binary bytes while a route is registered: one that lets the
request go on, one that replays it with route.fetch(), and one that
matches another URL. The server must receive the exact bytes and the
worker the response. A GET, and a 307 of a 64 KB POST whose second hop
must carry the route's headers, are covered too.

Firefox passes. On Chromium every POST of 64 KB or more never settles,
and one of 1 KB now and then: 13 of 20 cases fail per run, in every
run. The next commit fixes the cause.
With a route registered, a fetch from a dedicated worker could hang for
good on Chromium: every POST of 64 KB or more, one of 1 KB now and then.

A dedicated worker has no Fetch domain. Chrome pauses its requests on
the page's CDP session and announces them (Network.requestWillBeSent)
on the worker's, in either order. The chromium-bidi mapper keeps a
request on the target whose session sent its latest event and asks that
target's subscriptions and intercepts whether the request is held; a
worker's target is under none that names a user context or a tab. A
request paused first is on the worker's target once the mapper has both
events, so the mapper expects no intercept, waits for the request's
ExtraInfo, which Chrome sends only once the request is continued, and
neither reports the request nor continues it. A CDP dump shows the
fixups hand both events on at once, the mapper log "Request 64427.3 was
moved from <page target> to <worker target>", and nothing after it.

The fixups kept a request per session, so they could not tell that the
pause on one session and the announcement on another were one request.
A request is now kept once, by its id, with the sessions that report
it, and dedicated workers' sessions are known from
Target.attachedToTarget. A pause the mapper has yet to answer when a
worker's session first announces its request is handed to the mapper
again after the announcement: the request is back on the page's target
with both events, and the mapper reports it blocked. The trigger is the
pair of events itself, no timer, size, or retry.

The same model fixes what a worker's redirect hit once its first hop
went through: the hold of an early redirect pause counted announcements
on the pause's own session only, so a 307 of a 64 KB POST was continued
without the route's headers and reported as the response of its second
hop (5 of 5). The pause now waits for the hop the worker's session
announces, and the next hop's response ExtraInfo for that redirect.

A session that detaches takes what it paused and the requests no other
session reported; pauses another session holds for a hop it was to
announce go to the mapper instead of waiting for good.

tests/integration/route-worker.test.ts on Chromium: 13 of 20 cases
failed in 3 of 3 runs before, 20 of 20 pass after.
Plain connect to an Electron app failed with the raw 'browser.createUserContext: unknown error: Failed to create browser context.' as ENGINE_FAILURE. It is UNSUPPORTED_CAPABILITY now, with the browser's words and the option that drives such a host. Run on Electron 45.3.3: plain connect fails with it, context 'existing' passes.
4493caf dropped the half second after the two animation frames, taking the recording to be the page's repaints. Firefox's recorder lags them: the last frame of a segment held no ink in 7 of 41 runs (5 of 32 in a side by side run on 0f4e261 and d806dbe). With the wait back: 24 of 24.
@okwasniewski

Copy link
Copy Markdown
Member Author

[claude-fable-5-2] responding on behalf of okwasniewski

@kdy1 thanks for both reviews. The six inline threads have their own replies. The findings that had no thread, by review:

First review

Finding Status
1. route.fetch() and non-text bodies on Chromium Fixed in 1f55fd3: the body is read from the pause's postDataEntries. Rerun: Uint8Array [ff fe 00 80 41] arrives as fffe008041, new Blob(['BLOBDATA']) as BLOBDATA, a FormData with an in-memory File whole. A file from disk inside a form has no bytes in CDP, so route.fetch() throws ACTION_FAILED instead of sending none (Limits table); route.continue() still sends it. Test: tests/integration/route-fetch-bodies.test.ts.
2. e2e init CI templates and the Playwright cache Fixed in acef116: templates, docs/ci/gitlab.mdx, docs/ci/vercel-previews.mdx and the test cache E2E_BROWSERS_PATH, inside the project on GitLab.
3. Changeset and engine reference Fixed in 404aa7b and 19f996e: the changeset describes createResolver and createTextMatching as they are, engine.mdx documents them and ResolverPlatform, and urlMatches shows baseHref: string | undefined. MATCHES_TEXT_SOURCE is gone.
4. Text typed into an unfocused window Fixed in 41d8728: the driven page keeps focus (focus emulation on Chromium, the focus manager's test mode on Firefox), and text that would be entered in part fails with ACTION_FAILED. Rerun headed in a window that is never frontmost: café lands whole on both browsers.
Browser-open timeout as CANCELLED Fixed in 4aa9502. It is never CANCELLED now. With the default 60 s launchTimeout the runner's LAUNCH_TIMEOUT fires at the same moment and is what a run shows.
Missing changesets for two core changes Added in 19f996e: .changeset/core-standalone-secrets.md and .changeset/web-cookies-without-app-url.md.
Docs that still describe Playwright or WebKit Fixed in 3bbb1bc and 95a6b90.
A 304 never settles its record Fixed in 3789ce8 and 3256497: only 301, 302, 303, 307 and 308 with a Location lead to another hop. Tests: tests/integration/last-hop-statuses.test.ts, tests/unit/network-last-hop.test.ts.
20 MB cap, \r, mouse.move outside the viewport The cap and the pointer bounds are in the changeset and web.mdx (19f996e). \r presses Enter again (4793f7d), so there is nothing to name.
installSystemDeps under the cache lock Fixed in a624c3b and aa385a4: apt runs through async spawn, outside the lock.
Extra pointerUp after a drag's drop on Chromium Fixed in 2145601: the button is released before the synthetic drop. Rerun: pointerdown, mousedown, dragstart, pointercancel, dragenter, dragenter, drop, dragend, nothing after.
windows job on windows-latest No workflow here names a Blacksmith Windows label, so AGENTS.md now states the exception (4333262).

Second review, the two smaller items

Finding Status
App log drops the browser's own lines, objects read object Fixed in b4b8a17 and da10492: Chromium's log is read off CDP. Rerun: the WebSocket connection to ... failed line is back, and a logged object reads {code: 1006, reason: "", wasClean: false, nested: Object}, an event CloseEvent {isTrusted: true, wasClean: false, code: 1006, ...}. On Firefox a logged event still reads object (Limits table).
Agent and MCP screenshots and animations c346f1c and .changeset/web-screenshots-stop-animations.md: every screenshot, the agent's and MCP's included, stops CSS animations, transitions and element.animate() while it is taken. One that is paused is left as the page set it, so your CompanyOS case still reads "Screen unchanged"; the changeset says so.

Found while rerunning your repros, also fixed

  • A routed fetch from a dedicated worker could hang on Chromium when Chrome paused the request before announcing it (be61430, tests/integration/route-worker.test.ts).
  • body() of a waited 304 took 2 to 4 s on Firefox (42f5a48).

Left for Oskar to answer

  • Whether the breaking changes in .changeset/web-bidi-only.md were confirmed with the users who depend on them (WebKit, surfaceOf, reconnectEndpoint, clearState with an attempt-scope provider, the Firefox limits).
  • Whether the slower Firefox cold launch (+66%) is acceptable.

@okwasniewski
okwasniewski requested a review from kdy1 November 17, 2026 13:34
Node 22.22.3 ships undici 6.24.1, whose WebSocket fails the driver twice:

- It always offers permessage-deflate, Chrome takes it, and a message that inflates past 4 MB closes the connection (4.0 MB passes, 4.3 MB does not). A 6 MB response body, a 6 MB value from a page call, or an 800 KB binary POST under a route ended the attempt with 'the chromium-bidi mapper exited (code 0)'.
- Its mask buffer is 16386 bytes, so every frame after a process's 4096th is masked with zeros. Firefox closes a connection that sends one, then every new one: a probe got 4096 answers and lost the socket at command 4098, and check-actions.test.ts failed 3 of 3 runs.

Both browser sockets now use ws with compression off (browser-socket.ts). The mapper gets each CDP message in a task of its own, as it did from the old client's inflate step: handed a network read's messages in a row, it failed the attach tests 3 of 3.

On Node 22.22.3 and Node 24: the whole web suite passes on both browsers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants