Regression: parallel fanout invalidates a freshly reconnected Lab daemon lease

[chubes4]

Problem

A controller and Lab runner aligned on the same current release could not admit a four-cell Cook fanout immediately after a successful refresh-homeboy --reconnect.

The environment was explicitly converged first:

homeboy upgrade --force --skip-runners
homeboy runner refresh-homeboy homeboy-lab --ref v0.304.0 --reconnect

Both controller and active daemon reported homeboy 0.304.0+3e0ece8b6f14. The refresh returned daemon_refreshed: true, reconnect_required: false, and success.

A collision-safe custom fanout then dispatched four unique children. During admission Homeboy reported:

Lab offload: runner homeboy-lab is connected via direct SSH mode.
Lab offload: direct SSH runner homeboy-lab is not connected; attempting connection.

The outcomes were:

cook-issue-9621-v2: selected runner homeboy-lab cannot accept jobs
cook-issue-9623-v2: selected runner homeboy-lab cannot accept jobs
cook-issue-9624-v2: runner refused Lab admission reservation: daemon lease is not fresh
cook-issue-9626-v2: selected runner homeboy-lab cannot accept jobs

One cell reached remote preacceptance and its run-scoped workspace was materialized; all four terminalized as pre_execution_failure before provider execution.

This reproduces the user-facing contract from closed #9466 on v0.304.0 and overlaps the concurrency failure class from closed #8626, but with a freshly refreshed identical-build daemon. It is not #9626, which concerns dispatch through a known stale command binary.

Impact

A one-command fanout turns into daemon/session repair, unique rerun-ID management, and abandoned pre-execution recipes. The operator cannot trust successful refresh/reconnect as readiness evidence.

Expected contract

refresh-homeboy --reconnect returns success only when the selected daemon lease and admission endpoint are ready for the next bounded parallel fanout. Concurrent child admission shares one authoritative session and cannot invalidate it underneath sibling cells.

Acceptance criteria

• Reproduce refresh/reconnect followed immediately by at least four parallel Cook child admissions.
• Keep one authoritative direct-SSH session and daemon lease stable across all sibling reservations.
• Serialize or queue admission/reconnect ownership rather than allowing child commands to invalidate the shared session.
• Make refresh perform a bounded admission reservation readiness probe before returning success.
• Automatically recover transient daemon lease is not fresh and connection flips within the original child lifecycle.
• Do not consume provider budget or permanently strand a Cook recipe on preacceptance transport failure.
• Return one batch-level recovery action when automatic recovery is exhausted.
• Cover same-build reconnect and concurrent fanout, not only stale-binary replacement.

Evidence

• Batch: homeboy-orchestration-papercuts-20260722-v3
• Children: cook-issue-9621-v2, cook-issue-9623-v2, cook-issue-9624-v2, cook-issue-9626-v2
• Related closed regressions: Regression: refresh-homeboy succeeds before runner admissions are ready #9466 and Concurrent direct-SSH Lab commands invalidate the shared runner session #8626
• Related stale-binary guard: Lab handoff must not dispatch through a known stale runner command binary #9626
• Original downstream verification: Preserve university fixture structure and visual tokens Automattic/blocks-engine#659

AI assistance

• AI assistance: Yes
• Tool: OpenCode
• Model: OpenAI GPT-5.6 Sol
• Used for: Converged controller/runner versions, executed the parallel fanout, captured admission failures, deduplicated prior lease issues, and drafted this issue. Chris reviewed and owns the report.

[chubes4]

The first attempt to Cook this issue reproduced it exactly:

• Batch cook-homeboy-dogfood-9680-9682-20260722 created three clean issue worktrees from origin/main.
• All three children (cook-issue-9680, cook-issue-9681, cook-issue-9682) failed in lab_handoff_preacceptance with runner homeboy-lab refused Lab admission reservation: daemon lease is not fresh.
• No provider budget was consumed.

Local fallback attempts also exposed provider-path blockers:

• OpenCode run cook-issue-9682-local-1-attempt-1-737a86a5 auto-rejected access to its own controller scratch path and returned an empty patch.
• Codex run cook-issue-9682-local-2-attempt-1-530d02ca failed because codex.agent-task-executor returned malformed JSON: missing field uri at line 21 column 5.

The worktree /Users/chubes/Developer/homeboy@fix-issue-9682-homeboy remains clean. No candidate, commit, push, or PR exists after exhausting Lab, OpenCode-local, and Codex-local paths.

[chubes4]

New owning-layer blockers from local fallback are now tracked:

• Regression: OpenCode auto-rejects its assigned suffixed Cook scratch workspace homeboy-extensions#2350: OpenCode reconstructed the wrong controller-scratch allow path and auto-rejected its assigned workspace.
• Codex agent-task executor emits schema-invalid evidence refs without uri homeboy-extensions#2351: Codex emitted schema-invalid evidence refs without required uri.
• Run Cook verification against an immutable candidate checkout #9689: Cook verification runs in the dirty promotion target, making the repository review umbrella unusable before finalization.

These explain why no local fallback candidate could be finalized after the Lab lease reproduction.

[chubes4]

Additional production reproduction from the WP Build release gate on Homeboy 0.304.0+3e0ece8b6f14:

Trigger

homeboy review lint site-generation-agents --path /Users/chubes/Developer/wp-build@main/plugins/host/site-generation-agents --changed-only --summary
homeboy review lint site-forge --path /Users/chubes/Developer/wp-build@main/plugins/host/site-forge --changed-only --summary

Both Lab admissions failed with:

runner `homeboy-lab` refused Lab admission reservation: daemon lease is not fresh

Recovery attempted

homeboy runner refresh-homeboy homeboy-lab --ref v0.304.0 --reconnect
homeboy runner disconnect homeboy-lab

refresh-homeboy reported success, but subsequent admissions still failed freshness. disconnect then refused recovery because the controller retained many stale daemon generations whose persisted lease IDs differ from the authoritative active daemon lease.

Current homeboy runner status homeboy-lab evidence:

• generations.total: 34
• generations.draining: 33
• authoritative session lease: 23f27333-86d6-4292-beca-46d148aeeaaf
• authoritative daemon PID: 1768137
• daemon_freshness.fresh: false
• daemon_freshness.stale_reason_code: binary_hash_mismatch
• one legitimate active runner job prevented safe rotation

The failed disconnect iterated stale sessions and repeatedly reported persisted leases that did not match a different authoritative live lease, retaining the sessions and ledger instead of converging them.

User-visible impact

A fully passing WP Build behavioral/type/syntax/package verification cannot complete its configured fail-closed Lab lint gate. Refresh claims success without restoring admission, and safe disconnect cannot converge the retained generations while unrelated durable work is active.

Expected

A successful refresh/reconnect should atomically establish one admission-owning generation, retire superseded idle generations, and preserve legitimate active work without allowing stale persisted sessions to poison subsequent admissions. If convergence is deferred by active work, refresh should report that incomplete state rather than success.

[chubes4]

Reproduced again on homeboy 0.304.0+3e0ece8b6f14 while dispatching two independent agent-task cook --placement auto commands concurrently.

Both preflights reported homeboy-lab connected and the same controller/daemon build, then both failed at lab_handoff_preacceptance with runner homeboy-lab refused Lab admission reservation: daemon lease is not fresh. Both durable outcomes recorded provider_executions_consumed: 0.

Runs:

• agent-task-1d490711-7af8-45ce-b0e8-3a7483b23f49-attempt-1-49c3f690 for Keep Cook and agent-task status payloads compact by default #9676
• agent-task-1c33bd40-9992-4744-8965-9260a7a0923a-attempt-1-c10cd56e for Give deterministic Cook gates a declarative isolated environment #9677

This also shows auto selected the stale admission path and classified the terminal failure as non-retryable invalid_input, forcing an operator-directed local retry despite zero provider work.

AI assistance: OpenCode with OpenAI GPT-5.6 Sol was used to execute the Cooks, inspect durable aggregates, and correlate this reproduction with the existing tracker. Chris reviewed and owns this evidence.