[fix](VA) Keep in-progress playwright-cli commands in computer-use recordings - #14544
Conversation
Shell commands never enter the recording timeline, so browser automation driven via playwright-cli was hard-cut from computer-use videos by the post-stop smart trim. Open a recording action group when a playwright-cli command starts and commit it when the command finishes (including via a later ReadShellCommandOutput poll for long-running sessions), so that on-screen work is kept. Other shell commands remain trimmed as before. Co-Authored-By: Oz <oz-agent@warp.dev>
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR adds recording action groups around playwright-cli shell commands so Linux computer-use smart trimming can retain browser automation work, with a detector test and a helper to commit pending groups from later shell-output polling.
Concerns
- Long-running
playwright-clicommands that are still in progress when recording finalization runs remain only inpending_group, but finalization publishes only committedactions; this can still produce no artifact for a recording containing only that command, or trim the command window when earlier actions exist. - This is a user-facing recording-output change, but the PR description does not include a Linux recording or screen recording demonstrating the end-to-end behavior. Please attach a short recording showing a
playwright-clicommand preserved in the persisted computer-use video.
Verdict
Found: 0 critical, 2 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
A group left pending by a long-running playwright-cli command could reach finalization without ever being committed, so its window was trimmed. Commit any open pending group with the recording's current elapsed time when the active recording is claimed for finalization, sharing the settle logic used by begin_action_group's auto-commit and commit_action_group_now. Co-Authored-By: Oz <oz-agent@warp.dev>
vkodithala
changed the title
danielpeng2
left a comment
danielpeng2
left a comment
There was a problem hiding this comment.
left a couple non-blocking comments
| // computer-use recording: open an action group before the command | ||
| // starts so the smart cut does not trim its on-screen work. | ||
| let conversation_id = input.conversation_id; | ||
| let is_playwright_cli = is_playwright_cli_command(command); |
There was a problem hiding this comment.
not a now thing, but i wonder if we want to let the agent decide which actions should be visible in a recording. hardcoded detection of the playwright CLI seems fine, but if the agent ever uses any other command to control the desktop, that'd be missed.
one idea is to augment the run_command tool we provide to the agent with a is_meaningful_action parameter or something like that
There was a problem hiding this comment.
Yeah, I think this is a fair point. Probably OOS here but something we should consider.
Move the playwright-cli detection and the decision to open a recording action group out of ShellCommandExecutor and into RecordingController::maybe_begin_action_group, so the executor no longer leaks a recording implementation detail. Co-Authored-By: Oz <oz-agent@warp.dev>
2 participants
Description
On Linux, computer-use video recordings are smart-trimmed after stopping: only committed recording action groups produce keep-windows, and everything else is hard-cut. Shell commands never enter the recording timeline, so when the agent drives the browser via
playwright-cli, all of that on-screen work was cut from the video.This PR keeps in-progress
playwright-clicommands in the recording:ShellCommandExecutornow detectsplaywright-cliinvocations (skipping leading env-var assignments and resolving the program path's file name, sonpm install playwright-cliorecho playwright-clidon't match) and opens a recording action group before the command starts.playwright-clicommand that outlives the executor's poll window returns a long-running snapshot; the group stays open and is committed when a laterReadShellCommandOutputpoll observes the finished block. Since that poll runs in a separate executor call, a newRecordingController::commit_action_group_nowhelper commits the pending group using the recording's own elapsed clock (no-op when no recording is active or no group is pending).playwright-clisession never observed finished), claiming the active recording for finalization settles the pending group at the stop point instead of dropping it, via a sharedActiveRecording::commit_pending_group_nowhelper also used bybegin_action_group's auto-commit.Other shell commands (npm/cargo/etc.) never open a group, so they remain trimmed exactly as before. Behavior is unchanged when no recording is active: all recording calls no-op unless a recording is Active for the conversation.
Linked Issue
None — recording-quality fix scoped and dispatched via Oz orchestration.
ready-to-specorready-to-implement.Testing
Added a unit test for the
playwright-clicommand detector (detects_playwright_cli_commands) covering plain, env-prefixed, and absolute-path invocations plus non-matches (npm install playwright-cli,echo playwright-cli,cargo build).Added a regression test (
finalization_commits_open_pending_group) asserting that finalizing while a group is pending commits its window instead of dropping it.cargo nextest run -p warp -E 'test(detects_playwright_cli_commands) or test(block_working_directory_updated_does_not_drain_finish_senders) or test(/recording_controller/)'— 17/17 passed../script/format --check— clean.cargo clippy --workspace --exclude warp_completer --all-targets --tests -- -D warnings,cargo clippy -p warp --all-targets --tests -- -D warnings, andcargo clippy -p warp_completer --all-targets --tests -- -D warnings— all green.I have manually tested my changes locally with
./script/runRuntime behavior only differs on Linux (macOS recordings keep everything, no smart cut), so this was verified on macOS via build + unit tests rather than a live recording.
Agent Mode
CHANGELOG-BUG-FIX: In-progress playwright-cli browser automation is now preserved in computer-use video recordings instead of being trimmed out.
Co-Authored-By: Oz oz-agent@warp.dev