Skip to content

ci(docs): check the japanese docs against deepl - #1878

Draft
kacperkapusciak wants to merge 5 commits into
kacper/eng-1349-docs-japanesefrom
kacper/eng-1360-deepl-check
Draft

kacperkapusciak wants to merge 5 commits into
kacper/eng-1349-docs-japanesefrom
kacper/eng-1360-deepl-check

Conversation

@kacperkapusciak

@kacperkapusciak kacperkapusciak commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

What and why

Gives every translated docs section a second opinion from DeepL, so a mistranslated or stale section gets flagged in review.

  • scripts/check-docs-translations-deepl.ts splits docs/ja/<path>.mdx and docs/<path>.mdx at their headings, which pair up by position because check-docs-translations.ts keeps the outlines equal. For each section a pull request changes, in either language, DeepL translates our Japanese back into English and the English into Japanese.
  • scripts/deepl-review.ts scores both with chrF (character n-grams, so no Japanese tokenizer). A section is flagged when it is far from DeepL in both directions, left in English, or missing a glossary term that DeepL's version uses. A correct translation that only words things differently usually stays close in one direction.
  • The table in docs/ja/README.md is uploaded as a DeepL glossary (EN→JA and JA→EN), so DeepL uses our terms (ホスト型ブラウザ, サーバー). It is reused by a content hash, so an unchanged glossary costs nothing.
  • A section is flagged when both scores are below 0.25. That cut-off comes from the first real run on the site's UI copy: the one real mistranslation scored 0.11 at best, while rewordings scored up to 0.44.
  • A flag is a warning on the section's heading plus a side-by-side row in the job summary (English, DeepL's English, ours, DeepL's Japanese, scores). Every reviewed section, with its scores, is also uploaded as the deepl-report artifact, since GitHub shows only 10 warnings per step. The summary ends with the month's DeepL usage.
  • It never fails the job on a flag: DeepL is a second opinion, not the source of truth. A used-up monthly quota skips the review with a warning. The job fails only when DeepL can't be reached.
  • .github/workflows/docs-translations.yml runs it on pull requests that touch the docs. It is advisory and not a required check. It needs a DEEPL_API_KEY secret (Free plan, :fx key); without one, as on a fork, it skips.
  • Only changed sections are sent. The whole Japanese docs are about 1M characters, which is the Free plan's whole month. A full quickstart is about 8k.
  • Fallback. When DeepL's monthly quota runs out, the same review continues with openai/gpt-6-luna on the Vercel AI Gateway (AI_GATEWAY_API_KEY, already a repo secret). The model gets the glossary and the です・ます style in its prompt and answers a JSON list, one translation per section. The report and warnings name whoever translated. Without a gateway key, a used-up quota still skips the review with a warning.

Stacked on #1875. e2e-inc/e2e-site checks its UI copy the same way.

Issue

ENG-1360

Verified

Ran it locally: yes

  • Fallback, live: with DeepL stubbed to answer 456, gpt-6-luna reviewed the 9 quickstart sections in 23 s. Scores ranged from 0.65 to 1.00, and the only flags were 2 glossary ones.
  • node --test scripts/deepl-review.test.ts: chrF, glossary parsing (including the real README's 35 terms), the review rules, MDX section splitting, the real quickstart pairing up section by section, and the DeepL client (glossary created once and then reused, request shape, a 456 error surfaced).
  • Ran the script against a stubbed DeepL: warnings land on docs/ja/quickstart.mdx at each section's heading line, and --base HEAD sends nothing.
  • Ran against the real DeepL API with the repo's key, over every translated section (1,191, before it was limited to changed sections). It worked end to end, and used up the month's quota.
  • A used-up quota, stubbed with a 456, ends the job with a warning and exit 0.
  • pnpm docs:check, oxlint scripts/, and tsc -p scripts pass.

View guided diff Turn on auto-fix

Each changed section of a docs/ja page is translated back into English and compared with the English page's section, and compared with DeepL's own Japanese for the English, both with chrF. A section far from DeepL in both directions, left in English, or missing a glossary term DeepL used gets a warning on its heading and a row in the job summary. Sections pair by position, which check-docs-translations.ts guarantees. The glossary in docs/ja/README.md is uploaded as a DeepL glossary.
@github-actions github-actions Bot added the size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 93 passed

All 93 tests in 42 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 1.4s
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 1.1s
🟢 aria hidden flow › another plan shows the error and keeps the picker open 316ms
🟢 tests/async-states.e2e.ts · 2 passed 7.4s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.4s
🟢 async states › a wrong code is rejected after the toast has gone 5.0s
🟢 tests/canvas-only.e2e.ts · 2 passed 595ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 338ms
🟢 canvas only › a shape out of order paints the error and resets progress 257ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 745ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 383ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 362ms
🟢 tests/cart-totals.e2e.ts · 2 passed 732ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 419ms
🟢 cart totals › a quantity never drops below one 313ms
🟢 tests/checkout-review.e2e.ts · 1 passed 337ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 337ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 1.4s
🟢 context menu trap › right-click opens nothing; the row menu renames the file 623ms
🟢 context menu trap › the decoy menu items refuse and keep the file 789ms
🟢 tests/control-inventory.e2e.ts · 8 passed 6.7s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 677ms
🟢 control inventory › check picks a radio the pick replaces with its summary 505ms
🟢 control inventory › a long press and a double click are told apart from a tap 1.6s
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 591ms
🟢 control inventory › setInputFiles attaches files named from the project root 369ms
🟢 control inventory › the details view is left through the browser history 399ms
🟢 control inventory › scrollIntoView brings the footnote into view 451ms
🟢 control inventory › every exercise done shows the success message 2.1s
🟢 tests/css-content-ui.e2e.ts · 2 passed 1.2s
🟢 css content ui › the tile labelled Ship it advances and confirming ships 723ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 452ms
🟢 tests/date-picker.e2e.ts · 2 passed 1.6s
🟢 date picker › navigates three months ahead and books the required date 1.0s
🟢 date picker › confirming without the exact date shows the error 600ms
🟢 tests/debounced-search.e2e.ts · 2 passed 3.4s
🟢 debounced search › a decoy one character off is refused 1.7s
🟢 debounced search › picking the exact item adds it to the cart 1.7s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.4s
🟢 one submit opens one confirm, even when double-clicked 1.4s
🟢 tests/div-soup.e2e.ts · 2 passed 1.5s
🟢 div soup › a wrong code is rejected 676ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 783ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 1.1s
🟢 drag and drop › an out-of-order item resets the dropzone 546ms
🟢 drag and drop › dropping Banana then Cherry passes 533ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 551ms
🟢 filter deep link › filtering writes the query to the URL 320ms
🟢 filter deep link › opening a shared link restores the filter 231ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 1.1s
🟢 gift card purchase › refuses to buy without an amount 325ms
🟢 gift card purchase › refuses a recipient without an at sign 430ms
🟢 gift card purchase › sends the gift card 386ms
🟢 tests/hover-menu.e2e.ts · 2 passed 817ms
🟢 hover menu › a decoy leaf is refused 363ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 454ms
🟢 tests/iframe-form.e2e.ts · 2 passed 1.1s
🟢 iframe form › a wrong code is refused by the host page 565ms
🟢 iframe form › the code read from one frame applies through the other 535ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 871ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 214ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 316ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 341ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 3.2s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 3.2s
🟢 tests/kanban-board.e2e.ts · 3 passed 1.3s
🟢 kanban board › pointer drags move both cards into the goal layout 585ms
🟢 kanban board › a swipe along a path drags a card as well 312ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 422ms
🟢 tests/login-form.e2e.ts · 4 passed 1.5s
🟢 login form › rejects a malformed email 342ms
🟢 login form › rejects a short password 355ms
🟢 login form › rejects an unknown account 349ms
🟢 login form › signs in and logs out 411ms
🟢 tests/lying-labels.e2e.ts · 4 passed 1.3s
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 349ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 218ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 373ms
🟢 lying labels › visible text drives the payment through 399ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 918ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 319ms
🟢 native dialogs › dismissing the confirm cancels the order 296ms
🟢 native dialogs › a wrong gift code is rejected 303ms
🟢 tests/new-tab-flow.e2e.ts · 2 passed 956ms
🟢 new tab flow › reads the code in the tab the app opens and enters it back in the original tab 474ms
🟢 new tab flow › a tap that opens a tab leaves the screen on the original tab, and activate shows the new one 482ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.5s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.5s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 7.5s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 534ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 657ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 2.0s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 2.2s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 2.1s
🟢 tests/order-history.e2e.ts · 2 passed 1.6s
🟢 order history › overview and settings render their panels 287ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.4s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.7s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 594ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 274ms
🟢 otp auto-advance › a wrong code clears every box and starts over 879ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 690ms
🟢 continues only after the banner and the overlay are gone 690ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 832ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 393ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 439ms
🟢 tests/price-sorting.e2e.ts · 3 passed 642ms
🟢 price sorting › opens in the featured order with sale prices struck through 187ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 212ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 243ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.2s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.2s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.5s
🟢 buys a deal once the promo chrome is out of the way 1.5s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 839ms
🟢 rich text editor › bolding exactly the required word passes the check 408ms
🟢 rich text editor › bold that spills into the previous word fails the check 431ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 897ms
🟢 saved addresses › adds an address and clears the form 306ms
🟢 saved addresses › ignores an add with an empty field 276ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 315ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 698ms
🟢 shadow DOM form › rejects a wrong access code 356ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 342ms
🟢 tests/smoke.e2e.ts · 1 passed 405ms
🟢 home lists the scenarios and opens one 405ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 300ms
🟢 accepts the terms once the button clears the sticky footer 300ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 21.6s
🟢 scrolls the windowed table to the Golden Row and claims it 21.6s
🟢 tests/wishlist.e2e.ts · 2 passed 1.7s
🟢 wishlist › saving toggles the button and the tab badge 318ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.23.1 · 1m 31s · web · run artifacts

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 34 passed

35 agent steps · 26 replayed from cache · 25 model calls · 119.5k tokens (94% cached) · $0.0048

All 34 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 11.1s
🟢 control inventory › check sets the checkbox and the radio 1 step 1.7s
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 603ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 1.3s
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 3.1s
🟢 control inventory › upload attaches the named project files 1 step 3.0s
🟢 control inventory › back returns from the details view to the inventory 1 step 778ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 646ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 938ms
🟢 act signs in with the benchmark account 1 step 938ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 13.6s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 6.7s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 2.2s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 4.7s
🟢 tests-agent/scenarios.e2e.ts · 22 passed 22 steps · 17 calls 1m 23s
🟢 act completes shadow-dom-form 1 step · 3 calls 6.0s
🟢 act completes lying-labels 1 step · 3 calls 4.0s
🟢 act completes iframe-form 1 step · 3 calls 4.6s
🟢 act completes infinite-scroll 1 step 4.2s
🟢 act completes virtualized-table 1 step 22.0s
🟢 act completes overlay-trap 1 step 881ms
🟢 act completes sticky-chrome 1 step 904ms
🟢 act completes async-states 1 step · 3 calls 6.6s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 535ms
🟢 act completes drag-and-drop 1 step 534ms
🟢 act completes kanban-board 1 step 685ms
🟢 act completes context-menu-trap 1 step 2.7s
🟢 act completes stale-dom 1 step 497ms
🟢 act completes otp-auto-advance 1 step 739ms
🟢 act completes date-picker 1 step 811ms
🟢 act completes rich-text-editor 1 step 10.2s
🟢 act completes playbook-cleanup 1 step 613ms
🟢 act completes promo-storefront 1 step 2.3s
🟢 act completes new-tab-flow 1 step · 5 calls 9.1s
🟢 act completes gift-card-purchase 1 step 2.5s
🟢 act filters the list and the URL carries the filter 1 step 263ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 314ms
🟢 act adds an address with a run-unique label 1 step 314ms

Tested by e2e 0.23.1 · 1m 50s · web · run artifacts

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 34 model calls · 184.8k tokens (84% cached) · $0.0119

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 34 calls 6m 1s
🟢 act completes Login Form 1 step 29.9s
🟢 act completes Infinite Scroll List 1 step 21.8s
🟢 act completes Modal Flow 1 step 11.6s
🟢 act completes Bottom Tabs 1 step 7.9s
🟢 act completes Text Input Variations 1 step 7.7s
🟢 act completes Broken Accessibility 2 steps · 5 calls 14.2s
🟢 act completes Vision Only 2 steps · 1 call 6.5s
🟢 act completes Gestures 1 step · 9 calls 27.8s
🟢 act completes Async States 1 step · 8 calls 17.8s
🟢 act completes Debounced Search 1 step 7.5s
🟢 act completes Huge Virtualized List 1 step 1m 54s
🟢 act completes Flattened Registration Form 2 steps · 1 call 8.3s
🟢 act completes Flattened Login 2 steps · 6 calls 17.7s
🟢 act completes Sticky Chrome Target 1 step 9.4s
🟢 act completes WebView Accessibility 1 step · 4 calls 13.7s
🟢 act completes Permission Prompt 1 step 6.8s
🟢 act completes Product Catalog 1 step 9.0s
🟢 act completes Error Recovery 1 step 9.0s
🟢 act completes Choice Controls 1 step 9.0s
🟢 act completes Sequential Onboarding 1 step 11.2s

Tested by e2e 0.23.1 · 6m 2s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 56 passed, 1 skipped

All 57 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 3m 7s
🟢 control inventory › a label query resolves a labeled text field 15.2s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 16.8s
🟢 control inventory › a placeholder query and attribute reads find the field 4.7s
🟢 control inventory › a secure field takes a fill and denies value reads 8.1s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 9.8s
🟢 control inventory › toBeFocused follows the focus 8.9s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.0s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.6s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 13.5s
🟢 control inventory › a radio group reports the checked option 9.5s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.1s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.5s
🟢 control inventory › a header answers its accessible name 4.7s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.4s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.0s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.9s
🟢 control inventory › pointer verbs land at the point asked for 10.2s
🟢 control inventory › an ambiguous locator fails at once 5.8s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 12.2s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 193ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 12 passed 2m 0s
🟢 device fixture › setAppearance flips the color scheme the app reads 17.7s
🟢 device fixture › setOrientation rotates the window the app measures 9.4s
🟢 device fixture › the clipboard reads back what was written 9.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 7.2s
🟢 device fixture › closeApp then openApp relaunches on the home list 8.4s
🟢 device fixture › closeApp leaves the next commands on the same device 12.6s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 7.0s
🟢 device fixture › openLink opens the scenario a deep link names 14.7s
🟢 device fixture › device.back pops the scenario 5.9s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 4.7s
🟢 device fixture › setLocation and clearLocation change what the app reads 11.8s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 11.1s
🟢 tests/device.e2e.ts · 3 passed 49.8s
🟢 sequential onboarding echoes the exact values 27.3s
🟢 onboarding rejects a malformed email before advancing 10.0s
🟢 dismissing the keyboard over choice controls leaves them as they were 12.4s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 58s
🟢 infinite scroll reaches the target item 37.1s
🟢 product catalog adds exactly two of the right variant 10.3s
🟢 gestures: long-press, then swipe left 8.8s
🟢 sticky chrome: accept below the fold, then continue 13.0s
🟢 async states: load, pull to refresh, claim through the toast 12.0s
🟢 huge virtualized list reaches row 512 2m 25s
🟢 web view coupon form applies the code shown on the page 11.9s
🟢 tests/flows.e2e.ts · 7 passed 1m 11s
🟢 text inputs unlock the submit button 12.7s
🟢 filled text inputs report their values 11.0s
🟢 modal flow confirms through the native alert 9.8s
🟢 bottom tabs act inside the last tab 7.8s
🟢 error recovery retries, then confirms the delete 9.8s
🟢 choice controls place the exact order 7.7s
🟢 debounced search selects the target once results arrive 11.7s
🟢 tests/login-form.e2e.ts · 3 passed 45.8s
🟢 login form › rejects a malformed email 13.0s
🟢 login form › rejects an unknown account 13.2s
🟢 login form › signs in and logs out 19.6s
🟢 tests/smoke.e2e.ts · 1 passed 7.8s
🟢 home lists the scenarios and opens one 7.8s

Tested by e2e 0.23.1 · 6m 48s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 58 passed, 1 skipped

All 59 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 8s
🟢 control inventory › a label query resolves a labeled text field 28.9s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 14.4s
🟢 control inventory › a placeholder query and attribute reads find the field 8.6s
🟢 control inventory › a secure field takes a fill and denies value reads 8.4s
🟢 control inventory › press sends a character and Enter through the soft keyboard 8.8s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 9.4s
🟢 control inventory › toBeFocused follows the focus 9.5s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.4s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.1s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.2s
🟢 control inventory › a tab role answers getByRole 4.1s
🟢 control inventory › a radio group reports the checked option 7.7s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.1s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.4s
🟢 control inventory › a header answers its accessible name 4.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.1s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.1s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 7.6s
🟢 control inventory › a double tap lands inside the 300 ms window 6.5s
🟢 control inventory › an ambiguous locator fails at once 5.1s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.1s
🟢 refusals › verbs a device never declares are refused before the engine 3ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.7s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 24s
🟢 device fixture › setOrientation rotates the window the app measures 4.7s
🟢 device fixture › home leaves the app, openApp brings it back where it was 4.5s
🟢 device fixture › foregroundApp sees the home screen after home() 4.5s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.7s
🟢 device fixture › closeApp leaves the next commands on the same device 6.2s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.4s
🟢 device fixture › openLink opens the scenario a deep link names 4.5s
🟢 device fixture › device.back pops the scenario 4.8s
🟢 device fixture › unstable_raw reaches the test session through the raw driver 3.8s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.6s
🟢 device fixture › setLocation and clearLocation change what the app reads 23.9s
🟢 tests/device.e2e.ts · 5 passed 54.0s
🟢 permission prompt › allowing the system dialog enables the microphone 7.9s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 18.0s
🟢 sequential onboarding echoes the exact values 12.2s
🟢 onboarding rejects a malformed email before advancing 7.1s
🟢 dismissing the keyboard over choice controls leaves them as they were 8.8s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 40s
🟢 infinite scroll reaches the target item 32.2s
🟢 product catalog adds exactly two of the right variant 11.3s
🟢 gestures: long-press, then swipe left 9.2s
🟢 gestures: double-tap completes the flow 11.1s
🟢 sticky chrome: accept below the fold, then continue 14.6s
🟢 async states: load, pull to refresh, claim through the toast 15.7s
🟢 huge virtualized list reaches row 512 2m 6s
🟢 tests/flows.e2e.ts · 7 passed 55.8s
🟢 text inputs unlock the submit button 6.6s
🟢 filled text inputs report their values 5.8s
🟢 modal flow confirms through the native alert 9.2s
🟢 bottom tabs act inside the last tab 6.7s
🟢 error recovery retries, then confirms the delete 9.0s
🟢 choice controls place the exact order 8.4s
🟢 debounced search selects the target once results arrive 10.1s
🟢 tests/login-form.e2e.ts · 3 passed 28.3s
🟢 login form › rejects a malformed email 8.2s
🟢 login form › rejects an unknown account 7.5s
🟢 login form › signs in and logs out 12.6s
🟢 tests/smoke.e2e.ts · 1 passed 5.9s
🟢 home lists the scenarios and opens one 5.9s

Tested by e2e 0.23.1 · 10m 36s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 21 passed, 1 skipped

25 agent steps · 12 replayed from cache · 61 model calls · 396.4k tokens (83% cached) · $0.0262

All 22 tests in 1 file
Test Agent Time
⏭️ tests-agent/scenarios.e2e.ts · 21 passed, 1 skipped 25 steps · 61 calls 8m 47s
🟢 act completes Login Form 1 step 35.9s
🟢 act completes Infinite Scroll List 1 step 38.8s
🟢 act completes Modal Flow 1 step 12.9s
🟢 act completes Bottom Tabs 1 step 8.3s
🟢 act completes Text Input Variations 1 step 13.0s
🟢 act completes Broken Accessibility 2 steps · 6 calls 26.9s
🟢 act completes Vision Only 2 steps · 6 calls 28.3s
🟢 act completes Gestures 1 step · 11 calls 47.3s
🟢 act completes Async States 1 step · 10 calls 35.5s
🟢 act completes Debounced Search 1 step · 1 call 27.1s
🟢 act completes Flattened Registration Form 2 steps · 1 call 26.1s
🟢 act completes Flattened Login 2 steps · 11 calls 56.4s
🟢 act completes Sticky Chrome Target 1 step 13.2s
🟢 act completes WebView Accessibility 1 step 13.4s
🟢 act completes Permission Prompt 1 step 12.5s
🟢 act completes Photo Picker 1 step · 2 calls 31.6s
🟢 act completes Product Catalog 1 step 11.0s
🟢 act completes Error Recovery 1 step 9.9s
🟢 act completes Choice Controls 1 step · 5 calls 34.8s
🟢 act completes Stripe PaymentSheet 1 step · 8 calls 32.4s
🟢 act completes Apple Pay 1 step 11.8s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.1 · 8m 47s · ios-simulator · run artifacts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL Up to 999 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant