Repository navigation
docs(skills): bug bash repros from the finding, charters sized to the clock - #1601
Merged
Merged
Conversation
… clock Repro tests start as agent.act steps plus a closing agent.assert, with no locators; scripted hardening over MCP is opt-in. Each verifier runs its own repros into its own output and the report runs them all once. Confirmed means failing on the reported defect, read from the message for an act. Charters get ~15 min (was 6 steps), a time-limited bash keeps running new charters and reports each bug once confirmed, and short seeded passwords are called out.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Member
Author
|
[claude-opus-5-5] responding on behalf of okwasniewski Left for a human:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bundled skill's bug bash had a coding agent write every repro test with exact locators, found one by one through
e2e mcplocate, before a finding counted. That is the most expensive part of a bash and the one most likely to fail for reasons unrelated to the bug. Its charters also stopped at--max-steps 6, a number from when explore capped runs at 12 steps.agent.actsteps, one goal each, plus its expected behavior as the closingagent.assert. It needs no locators, and the charter's credential goes in as step parameters (or a setup test's session).ASSERTION_FAILEDfrom the closingassert, or a failedactwhose message names the defect. A failedactcarries the code the model chose, so the skill says to read its message and screenshot. A pass means the bug did not reproduce.--output .e2e/bugbash/verify-<area>, since a run clears its output's results. The report runs every repro once withnpx e2e run --tag bugbash --config e2e.bugbash.config.ts.screenactions, exactexpect, MCPlocate) is a step for when the user wants a regression test. A scripted step that opens a native dialog needsbrowser.onDialog.--max-steps 15 --timeout 900000, about fifteen minutes. Under a time limit the first round ends by a third of it, the bash keeps running charters for what no charter reached until a tenth is left, and each bug is reported once its repro confirms it.admin/admin) is called out before the config fails on it.docs/bug-bash.mdx,docs/reference/mcp.mdx, and the create-verification-skill generator and its example now describe the same bar.Verified
Ran it locally: yes
tsc --noEmit).npx e2e run --tag bugbashwithopenai/gpt-6-luna-fast: two real bugs failed withASSERTION_FAILEDnaming the defect, one from the closingassertand one from theactthat met it. The not-a-bug control passed. The three repros took 14.9 s and $0.0032 together.confirm()and leave-page dialogs. Without it, the confirm-gated repro endsAUTOMATION_UNSUPPORTEDand the control endsACTION_FAILED.