Skip to content

Classify benchmark gate failures separately from orchestration failures #9880

Description

@chubes4

Problem

A Lab-offloaded fixture benchmark can execute its orchestration successfully and produce complete structured evidence, then correctly fail its quality gate. The operator-facing failure path frames that outcome like remote execution failure and directs the operator toward raw output-file inspection instead of presenting the typed gate result and its evidence commands.

Run 705aad31-551f-4ab6-8db3-a6e17e903b15 demonstrates the distinction:

  • Lab handoff and child recipe completed.
  • Child recipe exit_code was 0.
  • Child output was parsed successfully.
  • The run retained benchmark results, summary, finding packets, screenshots, DOM snapshots, and visual reports.
  • Benchmark status was fail because failed_fixture_count was 1, with explicit runtime_execution_failed and visual_mismatch categories.

This is an expected workload gate rejection, not a transport, runner, or orchestration failure. Treating both as one generic remote failure sends operators into manual artifact spelunking and obscures whether retrying infrastructure would help.

Context: Automattic/blocks-engine#659 and child run 705aad31-551f-4ab6-8db3-a6e17e903b15.

Expected

Homeboy reports separate typed outcomes for:

  1. orchestration/transport failure,
  2. workload execution failure,
  3. completed workload with rejected result gate.

A rejected benchmark gate should summarize failed metrics/categories and provide direct commands for the result, summary, and finding artifacts.

Acceptance criteria

  • Runner and bench envelopes preserve a distinct result_gate_failed or equivalent outcome.
  • A child process exit 0 plus parsed benchmark result cannot be summarized as remote orchestration failure.
  • CLI output displays failed gate metrics and top typed failure categories.
  • CLI output provides resolvable homeboy runs artifact get commands for result, summary, and finding packets.
  • Retry guidance is limited to transport/orchestration failures; result-gate failures recommend evidence inspection or candidate repair.
  • Tests cover transport failure, nonzero workload exit, malformed result, and successful execution with a rejected result gate.

AI assistance

  • AI assistance: Yes
  • Model: OpenAI GPT-5.6 Sol
  • Tool: OpenCode
  • Used for: run inspection, outcome classification, duplicate search, and issue drafting

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions