fix(ios): give keyboard dismiss a safe-area-tap fallback (#1598) - #1606
Conversation
The runner already tapped a keyboard's own Hide/Dismiss/Done key when the AX tree exposed one, but iPhone's default software keyboard has no such key, so `keyboard dismiss` returned UNSUPPORTED_OPERATION on the common case and agents proceeded with the keyboard (and any live QuickType predictive-text bar) still up. Live-validated on throwaway simulators before choosing a design: hardware escape key (no effect without a connected hardware keyboard), swipe-down starting on the keyboard (does not trigger UIKit's interactive dismissal on Settings/Safari/Contacts), and a private `performAction:onElement:value:error:` AX call (hung the runner for 90s on a guessed action name, force-killed by the daemon timeout) were all ruled out. The dismiss-key tap remains the primary mechanism (iPad, or any app with an inputAccessoryView Done/Cancel button); a new snapshot-derived safe-area tap is added as the disclosed last resort, computed to land outside both the keyboard and every currently-hittable element so it is a safe no-op even when it fails to dismiss. The response now discloses which mechanism actually fired (`mechanism: 'dismissKey' | 'safeAreaTap'`) across the CLI/daemon dispatch path, the SDK runtime.backend surface, and session-event summaries, so callers can tell a real dismiss-key press apart from a best-effort tap. UNSUPPORTED_OPERATION now says both mechanisms were tried.
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
|
P1: The safe-area tap is not safe by construction. Exact-head Lint & Format and Fallow are also owner-action red. |
oxfmt on three touched files; buildKeyboardActionSummary split so the dismiss wording (incl. the safeAreaTap mechanism disclosure) lives in its own helper below the complexity threshold.
|
CI fixes pushed in 5676e81: oxfmt on the three touched files, and buildKeyboardActionSummary split so the dismiss wording (incl. the safeAreaTap mechanism disclosure) sits in its own helper under the complexity threshold — which also let fallow's inherited-finding exclusion correctly absorb the untouched handleSettingsCommand flag. Fallow now reports no issues in the 16 changed files; lint and format clean. 🤖 Addressed by Claude Code |
…eview P1) A role allowlist cannot prove a point is AX-empty: an unlabeled RN Pressable surfaces as a hittable Other, and a tappable parent can cover a point its static-text child does not. Every known element frame now counts as an obstacle regardless of role or hittability, with only ~window-sized structural frames exempt (isStructuralRootFrame, 95% coverage) — exempting those is what keeps the rule satisfiable, and a genuinely tappable full-screen backdrop staying exempt is the disclosed, accepted behavior of this fallback. One .any resolution replaces ten typed queries (single tree snapshot, no per-element isHittable round trips), so the stricter rule is also cheaper.
|
P1 addressed in 4deb5dc. The allowlist is gone: every known element frame is now an obstacle, regardless of role or hittability — one The one deliberate exemption, and why it must exist: ~window-sized structural frames ( New in-bundle test pins the boundary (roots and full-bleed frames exempt; banners, half-screen sheets, and small controls are obstacles); all 7 keyboard safe-area tests pass on a live simulator build. 🤖 Addressed by Claude Code |
|
P1 remains at current head Please distinguish inert Application/Window roots from actionable full-screen elements rather than inferring structure from geometry, or keep generic background-tap dismissal unsupported. Pin the production obstacle mapping with a full-screen actionable |
…#1606 review P1, round 2) No geometry or role query can prove a coordinate is side-effect-free: after the any-element rule, the structural-root exemption still deliberately removed full-screen actionable elements (RN Pressable backdrops) from the obstacle set, so the tap could navigate or submit — and report success because the mutation hid the keyboard. Per review, generic background-tap dismissal is now explicitly unsupported: the dismiss key is the only mechanism the runner vouches for, UNSUPPORTED_OPERATION says so and steers callers to press-the-next-target / keyboard enter, and the mechanism field narrows to 'dismissKey'. Unrecognized wire mechanisms degrade to the bare message and are dropped from event details.
|
Round-2 P1 accepted in full — resolved by taking the review's second option in a58640f: the safe-area tap is removed entirely; generic background-tap dismissal is now explicitly unsupported. The deciding logic: after the any-element rule, the only way to keep the fallback satisfiable was the structural-root exemption — and as you showed, geometry cannot distinguish an inert Window root from a full-screen actionable What remains: 🤖 Addressed by Claude Code |
|
1 participant
Important
Revised after review round 2: the safe-area tap fallback described below was REMOVED (
14ad…): background-tap dismissal is deliberately unsupported — no query can prove a coordinate side-effect-free.dismissKeyis the only mechanism; when absent,keyboard dismissreportsUNSUPPORTED_OPERATIONwith recovery guidance. The live matrix below stands as the evidence that motivated both the attempt and its removal.Summary
keyboard dismissalready tapped the keyboard's own Hide/Dismiss/Done key when the AX tree exposed one, but iPhone's default software keyboard has no such key, so the command returnedUNSUPPORTED_OPERATIONon the common case. Adds a new snapshot-derived safe-area tap as a disclosed last-resort fallback, computed to land outside both the keyboard and every currently-hittable element the runner knows about — a safe no-op even when it fails to dismiss.mechanism: 'dismissKey' | 'safeAreaTap', threaded through the CLI/daemon dispatch path (src/core/dispatch.ts), the SDKruntime.backendsurface (src/commands/system/runtime/system.ts), and session-event transcript summaries (src/daemon/session-event-action-presentation.ts).UNSUPPORTED_OPERATIONnow says both mechanisms were tried.Live-validation matrix
Before picking a design I live-tested every candidate mechanism against real apps on throwaway simulators (
ad-kbd-fixiPhone 17 Pro,ad-kbd-fix-ipadiPad Pro — both deleted, see Cleanup below), since this codebase's own gesture-viewport code (frameAvoidingKeyboardinRunnerTests+Interaction.swift) already deliberately avoids touching the keyboard region, which was a signal that touching the keyboard is not straightforward.First namefieldkeyboard dismissreportedKeyboard dismissed via its dismiss key, keyboard fully gone. Kept as the primary mechanism.app.typeKey(.escape, modifierFlags: []), public XCTest API)RunnerSynthesizedGesturecontinuous-drag bridge)UIScrollView.keyboardDismissModebehavior, not a system-wide guarantee — neither screen had it enabled.defaults write com.apple.iphonesimulator ConnectHardwareKeyboard)bench-golden-bsky-latest,ad-bsky-repro, anotheriPhone 17 Pro) — toggling it live could have altered their typing behavior. Rejected as a runner-side fix regardless: it's Simulator-only (no physical-device equivalent) and the runner process can't reach the host Simulator.app UI to toggle it itself.performAction:onElement:value:error:with a guessed action name (AXHideKeyboard)_accessibilityHideKeyboardselector and the AX client'sperformAction:onElement:value:error:method, so I tried invoking it — the call hung for the full 90s daemon timeout and the runner process was force-killed. App state was undamaged afterward, but a wrong guess on a private, undocumented action name costs a full timeout + forced runner restart per attempt. Rejected as too risky to ship without official documentation of the correct action name/target.RunnerKeyboardDismissSafeArea.safePointonly returns a point outside the keyboard and outside every obstacle frame — and (b) many RN screens do implementKeyboard.dismiss()/Pressable-wraps-screen on background tap, which this fallback correctly benefits from even though the specific stock apps tested here don't.Byte-exactness evidence (text preservation)
Addressing the benchmark corruption case directly: filled a multi-word string, then ran
keyboard dismissand re-snapshotted to confirm the field text is byte-exact afterward, on both the success and failure paths.hello world testinginto Contacts' First-name field →keyboard dismiss→Keyboard dismissed via its dismiss key→ re-snapshot showshello world testingunchanged, keyboard fully gone.hello from the benchmarkinto Contacts' First-name field (the coordinator's own repro string) →keyboard dismiss→UNSUPPORTED_OPERATION: Unable to dismiss the iOS keyboard: no dismiss key was exposed and a safe-area tap did not resign it→ re-snapshot showshello from the benchmarkunchanged — no corruption, no spurious appended word.I was not able to reproduce the exact QuickType-predictive-bar corruption from the bsky repro (bsky wasn't available in this environment, and Settings/Contacts' text fields don't surface a live predictive-completion bar the way the bsky compose view does), so I can't directly confirm "resigning does not commit a pending QuickType suggestion" for that specific UI. What I can confirm: the fix's failure path no longer lets an agent believe dismiss succeeded when it didn't — it now honestly reports
UNSUPPORTED_OPERATION, which is the behavior that would let an agent-side workflow choose not to publish immediately. Closing the loop on the QuickType-commit question specifically (doesresignFirstRespondervia the dismiss-key tap commit vs. discard a pending predictive suggestion) is filed as a follow-up — it needs the actual bsky compose view (or an RN fixture with a live predictive bar) to observe.Response shape
mechanismis threaded throughBackendKeyboardResult(src/backend.ts), the publicKeyboardCommandResultcontract (packages/contracts/src/keyboard.ts), and surfaced in session-event summaries/details ("Dismissed keyboard (safe-area tap)").Also updated
keyboard dismissandkeyboard --helptext (src/cli/parser/cli-help.ts,src/commands/system/index.ts) now describe the fallback and mechanism disclosure.apple/runner/RUNNER_PROTOCOL.mdexample error message updated to match.Test plan
RunnerKeyboardDismissSafeArea.safePoint(7 cases: clear center, obstructed-candidate fallback, all-candidates-obstructed → nil, keyboard-covers-screen → nil, empty window → nil, missing-keyboard-frame tolerance, obstacle-margin near-miss) — run viaxcodebuild test-without-buildingagainst a real simulator (AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1): 9/9 passed (7 new + 2 pre-existing regression checks selected from the same file).src/core/__tests__/dispatch-keyboard.test.ts(dismissKey/safeAreaTap/no-mechanism message+field cases),src/daemon/__tests__/session-event-action.test.ts(transcript summary/details disclosure),src/commands/system/runtime/system.test.ts(SDKruntime.backendsurface).pnpm check:quick(lint + typecheck across workspace) — clean.pnpm test:unit(full suite, run synchronously) — 603 test files / 5297 tests passed, 0 failures. (Two tests flagged by the slow-test gate as within the 2x load-variance band, not failing — pre-existing timing noise unrelated to this change:input-actions.test.tsfillAndroid retry,request-router-replay-scope.test.tsdevice retention.)UNSUPPORTED_OPERATION).resignFirstResponderdoes not commit a pending predictive suggestion.Cleanup
Both throwaway simulators created for live validation were deleted at the end of the session:
ad-kbd-fix(iPhone 17 Pro) andad-kbd-fix-ipad(iPad Pro). Confirmed viaxcrun simctl list devices— neither appears. No sessions, daemons, or state under this repo's default~/.agent-devicedir were left bound to them.