Repository navigation
fix(web): answer native dialogs an agent step's action opens - #1599
okwasniewski wants to merge 6 commits into
Conversation
An unhandled alert, confirm, or leave-page dialog after an agent step's action is accepted, a prompt dismissed, and the app log names it. Before, the engine dismissed it and failed the agent's next action, so leaving a dirty form or confirming a delete was out of reach for agent.act, explore, and MCP action tools. A dialog belongs to the newest action before it, never to a read, so a test's own dialog still fails its next step even while an agent judgment polls. OperationContext.agentStep marks agent step operations, replays included, so a recorded dialog step replays under --strict-cache; the web benchmark gains a deferred-dialog agent scenario with its recording.
🟢 e2e web: 91 passedAll 91 tests in 41 files
Tested by e2e 0.23.0 · 1m 4s · web · run artifacts |
🟢 e2e web agent: 34 passed35 agent steps · 26 replayed from cache · 23 model calls · 102.4k tokens (94% cached) · $0.0041 All 34 tests in 5 files
Tested by e2e 0.23.0 · 1m 24s · web · run artifacts |
There was a problem hiding this comment.
All reported issues were addressed across 25 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 40 model calls · 221.0k tokens (84% cached) · $0.0137 All 20 tests in 1 file
Tested by e2e 0.23.0 · 5m 44s · android-emulator · run artifacts |
🟢 e2e ios: 55 passed, 1 skippedAll 56 tests in 7 files
Tested by e2e 0.23.0 · 7m 14s · ios-simulator · run artifacts |
…and media changes An action guard refuses before it runs no longer claims the next dialog, and browser.setViewport and browser.emulateMedia claim it like other page changes. Docs: an unhandled dialog fails the attempt at its end when no step follows, a run_code dialog fails the next statement first, and the skill warns that agent steps accept confirms unseen without a handler.
🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped25 agent steps · 12 replayed from cache · 58 model calls · 397.8k tokens (83% cached) · $0.0262
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped | 25 steps · 58 calls | 8m 45s | |
| 🟢 | act completes Login Form | 1 step | 45.0s |
| 🟢 | act completes Infinite Scroll List | 1 step | 40.8s |
| 🟢 | act completes Modal Flow | 1 step | 10.9s |
| 🟢 | act completes Bottom Tabs | 1 step | 7.6s |
| 🟢 | act completes Text Input Variations | 1 step | 13.6s |
| 🟢 | act completes Broken Accessibility | 2 steps · 5 calls | 25.3s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 29.4s |
| act completes Gestures (1 failed attempt first) | 1 step · 9 calls | 38.8s | |
| 🟢 | act completes Async States | 1 step · 10 calls | 35.0s |
| 🟢 | act completes Debounced Search | 1 step · 1 call | 28.0s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 29.4s |
| 🟢 | act completes Flattened Login | 2 steps · 10 calls | 52.6s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 14.0s |
| 🟢 | act completes WebView Accessibility | 1 step | 12.9s |
| 🟢 | act completes Permission Prompt | 1 step | 12.0s |
| 🟢 | act completes Photo Picker | 1 step · 2 calls | 30.6s |
| 🟢 | act completes Product Catalog | 1 step | 10.4s |
| 🟢 | act completes Error Recovery | 1 step | 10.0s |
| 🟢 | act completes Choice Controls | 1 step · 6 calls | 36.5s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 8 calls | 31.7s |
| 🟢 | act completes Apple Pay | 1 step | 10.8s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
Tested by e2e 0.23.0 · 9m 21s · ios-simulator · run artifacts
🟢 e2e android: 57 passed, 1 skippedAll 58 tests in 7 files
Tested by e2e 0.23.0 · 10m 6s · android-emulator · run artifacts |
|
[claude-opus-5-5] responding on behalf of okwasniewski Left for a human (also under "For review" in the description):
|
There was a problem hiding this comment.
I think we should remove this file
There was a problem hiding this comment.
1 issue found across 24 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/web-benchmark/tests-agent/scenarios.e2e.ts">
<violation number="1" location="apps/web-benchmark/tests-agent/scenarios.e2e.ts:135">
P2: This scenario has no committed recording under `apps/web-benchmark/.e2e/cache/`. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with `test:agent` and commit the new cache entry in this PR.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| }, | ||
| { | ||
| // The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included. | ||
| slug: 'deferred-dialog', |
There was a problem hiding this comment.
P2: This scenario has no committed recording under apps/web-benchmark/.e2e/cache/. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with test:agent and commit the new cache entry in this PR.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/web-benchmark/tests-agent/scenarios.e2e.ts, line 135:
<comment>This scenario has no committed recording under `apps/web-benchmark/.e2e/cache/`. In CI the step will call the model instead of replaying, so the suite loses its deterministic, no-cost gate and fails on PRs without the key. Re-record it locally with `test:agent` and commit the new cache entry in this PR.</comment>
<file context>
@@ -130,6 +130,12 @@ const SCENARIOS: readonly Scenario[] = [
},
+ {
+ // The confirm opens 900 ms after the click, and the engine accepts it for the agent, replay included.
+ slug: 'deferred-dialog',
+ goal: 'submit the report and accept the confirmation that follows',
+ success: 'Report submitted after deferred confirmation',
</file context>
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
Requires human review: Auto-approval blocked by 1 unresolved issue from previous reviews.
View guided diff | Turn on auto-fix | Re-trigger cubic
An agent never sees a native
alert,confirm,prompt, or leave-page dialog: they sit outside the page. With nobrowser.onDialoghandler, the web engine dismissed each one and failed the next operation withINVALID_STATE. So an agent step could not leave a form with unsaved changes or confirm a delete, live or from the replay cache.DialogRouterasks its surface whether an agent step made the newest action (packages/web/src/dialogs.ts). If so, a dialog no handler takes is answered: alerts, confirms, and leave-page dialogs are accepted, and a prompt is dismissed, since the agent never sees its text. The app log gets onesystemline, such asconfirm dialog "Delete this label?" accepted for the agent.PlaywrightSurface.guardAction()(packages/web/src/surface.ts) wraps every operation that acts: navigation, input,browser.evaluate, keyboard, mouse,browser.setViewport,browser.emulateMedia. It claims the dialog once the operation runs, so one refused by a pending error claims nothing. Reads (observe,locate, screenshots) and a route handler'sroute.fetchnever take a dialog. So a test's own late dialog still fails its next step while an agent judgment polls, and a test route handler fetching mid-action can't steal the agent's dialog.OperationContext.agentStep(new, optional) marks an agent step's operations, replays included (packages/e2e/src/agent/phases.ts). Replayed actions keeporigin: 'test'for settling, so without it a recorded dialog step failed under--strict-cache. It crosses the engine protocol. An older core omits it and the engine falls back toorigin.INVALID_STATE. A registered handler always wins.docs/reference/web.mdxsaid an unhandled dialog throwsAPP_NOT_OPEN. The code throwsINVALID_STATE, so the table now says that./e/deferred-dialog, whose confirm opens 900 ms after the click, with its committed recording.Why
In 201
e2e exploreruns on 0.22.0 against a self-hosted Gitea (theirreport.jsonand AI traces):unhandled beforeunload dialog.navigatecalls failed withnet::ERR_ABORTED. 113 of them were the dismissed leave-page dialog.Action failed: unhandled beforeunload dialog:and retriedbackuntil the step timed out.For review
browser.onDialog.run_codeis test code, so its dialogs still fail the next call. Surfacing answered dialogs in action results is a follow-up.driver/dialogs.tsimports.BidiSessionbuilds the router there and would pass the same two callbacks, and its action entry points would takeguardAction's ownership line.Verified
Ran it locally: yes
packages/web/tests/integration/agent-dialogs.test.ts, real Chromium, 8 cases: an agent leaves a dirty form; agent confirm, alert, and prompt; a replayed agent action (origin: 'test',agentStep: true); a test route handler'sroute.fetchbefore the agent's alert; a test'ssetViewportwhose resize handler confirms after an agent tap, which must fail the test's next step; a test action refused by a pending route error, which must leave the agent's alert with the agent; a test's 300 ms delayed confirm while agent reads poll, which must still fail withINVALID_STATE; and a test's own confirm. On main, both live agent cases fail withunhandled ... dialog. The viewport and refused-action cases fail without their fixes.tests-agentdeferred-dialog: recorded live once, then replayed twice withCI=1 --strict-cache:Cache 1 replayed, no model call, and the app log showsconfirm dialog "Really submit the report?" accepted for the agent.--strict-cache, with the same counts on main for the existing recordings:deferred-dialog)tests/dialog.e2e.tsandtests/dialogs-frames.e2e.ts(15 tests) and web benchmarktests/native-dialogs.e2e.tsandtests/deferred-dialog.e2e.ts(4 tests) pass, so scripted handlers behave as before.e2e explore(openai/gpt-6-luna-fast,--max-steps 4) on a scratch notes app with a leave-page guard on its editor and aconfirm()before each delete. Its delete removes the wrong note, a planted bug:unhandled beforeunload dialog)e2e run --tag bugbash):AUTOMATION_UNSUPPORTED: blocked by the confirmASSERTION_FAILED: confirmedACTION_FAILED: blocked by the leave-page dialog