Problem
A Lab-offloaded fixture benchmark can execute its orchestration successfully and produce complete structured evidence, then correctly fail its quality gate. The operator-facing failure path frames that outcome like remote execution failure and directs the operator toward raw output-file inspection instead of presenting the typed gate result and its evidence commands.
Run 705aad31-551f-4ab6-8db3-a6e17e903b15 demonstrates the distinction:
- Lab handoff and child recipe completed.
- Child recipe
exit_code was 0.
- Child output was parsed successfully.
- The run retained benchmark results, summary, finding packets, screenshots, DOM snapshots, and visual reports.
- Benchmark status was
fail because failed_fixture_count was 1, with explicit runtime_execution_failed and visual_mismatch categories.
This is an expected workload gate rejection, not a transport, runner, or orchestration failure. Treating both as one generic remote failure sends operators into manual artifact spelunking and obscures whether retrying infrastructure would help.
Context: Automattic/blocks-engine#659 and child run 705aad31-551f-4ab6-8db3-a6e17e903b15.
Expected
Homeboy reports separate typed outcomes for:
- orchestration/transport failure,
- workload execution failure,
- completed workload with rejected result gate.
A rejected benchmark gate should summarize failed metrics/categories and provide direct commands for the result, summary, and finding artifacts.
Acceptance criteria
- Runner and bench envelopes preserve a distinct
result_gate_failed or equivalent outcome.
- A child process exit
0 plus parsed benchmark result cannot be summarized as remote orchestration failure.
- CLI output displays failed gate metrics and top typed failure categories.
- CLI output provides resolvable
homeboy runs artifact get commands for result, summary, and finding packets.
- Retry guidance is limited to transport/orchestration failures; result-gate failures recommend evidence inspection or candidate repair.
- Tests cover transport failure, nonzero workload exit, malformed result, and successful execution with a rejected result gate.
AI assistance
- AI assistance: Yes
- Model: OpenAI GPT-5.6 Sol
- Tool: OpenCode
- Used for: run inspection, outcome classification, duplicate search, and issue drafting
Problem
A Lab-offloaded fixture benchmark can execute its orchestration successfully and produce complete structured evidence, then correctly fail its quality gate. The operator-facing failure path frames that outcome like remote execution failure and directs the operator toward raw output-file inspection instead of presenting the typed gate result and its evidence commands.
Run
705aad31-551f-4ab6-8db3-a6e17e903b15demonstrates the distinction:exit_codewas0.failbecausefailed_fixture_countwas1, with explicitruntime_execution_failedandvisual_mismatchcategories.This is an expected workload gate rejection, not a transport, runner, or orchestration failure. Treating both as one generic remote failure sends operators into manual artifact spelunking and obscures whether retrying infrastructure would help.
Context: Automattic/blocks-engine#659 and child run
705aad31-551f-4ab6-8db3-a6e17e903b15.Expected
Homeboy reports separate typed outcomes for:
A rejected benchmark gate should summarize failed metrics/categories and provide direct commands for the result, summary, and finding artifacts.
Acceptance criteria
result_gate_failedor equivalent outcome.0plus parsed benchmark result cannot be summarized as remote orchestration failure.homeboy runs artifact getcommands for result, summary, and finding packets.AI assistance