Skip to content

Add runner-aware remote discovery to bench list #9623

Description

@chubes4

Problem

When a Lab benchmark reports discovered ids: <none>, Homeboy has no supported command to run the same scenario-discovery contract on the selected runner.

The obvious diagnostic command is rejected:

homeboy bench list static-site-importer \
  --path /path/to/worktree \
  --placement lab \
  --runner homeboy-lab
error: unexpected argument --placement found

tip: to pass --placement as a value, use -- --placement

homeboy bench list --help exposes --path, --extension, and --rig, but no --runner or --placement. Passing these after -- would send them to the workload rather than inspect discovery remotely, so the generated tip is not a valid remediation.

This forced manual runner-job log inspection while diagnosing browser verification for Automattic/blocks-engine#659.

Expected

bench list should support the same execution environment selection as bench and report what the selected runner can actually discover after materialization/hydration.

Suggested shape:

homeboy bench list static-site-importer \
  --runner homeboy-lab \
  --path /path/to/worktree \
  --rig static-site-importer-fixture-matrix

The output should include controller and runner workload identities, extension/runtime versions, hydration state, discovered IDs, and bounded discovery diagnostics.

Acceptance criteria

  • bench list accepts --runner and placement policy consistently with bench.
  • Local paths and path settings are materialized/remapped exactly as execution would do.
  • Discovery runs against the selected runner runtime and extension set.
  • The command remains read-only and does not execute benchmark workloads.
  • Unsupported-option guidance does not recommend forwarding Homeboy options through workload passthrough argv.

AI assistance

  • AI assistance: Yes
  • Tool: OpenCode with OpenAI GPT-5.6 Sol
  • Used for: Reproduced the CLI gap and drafted the operator contract with Chris.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions