Skip to content

Runner heartbeat-only jobs can block daemon rotation indefinitely #9702

Description

@chubes4

Problem

A runner.exec workload can remain running indefinitely when its wrapper continues emitting heartbeats even though the child workload makes no meaningful progress. Because the job remains active, a daemon with binary_hash_mismatch is not restartable, runtime generations cannot drain, and new Lab work is rejected with daemon lease is not fresh.

Observed evidence

  • Homeboy: 0.304.0+3e0ece8b6f14
  • Runner: homeboy-lab
  • Job ID: d911d948-062f-4ffb-b089-1671290d2177
  • Durable run ID: blocks-engine-663-standalone-demo-94c26a4d
  • Source workload: Improve Figma image projection and responsive layout Automattic/blocks-engine#663
  • Started: 2026-07-22T12:48:09.521Z
  • Still reported active after more than three hours
  • Process tree: 8 processes, about 657 MB RSS, only about 33 seconds cumulative user CPU after roughly three hours
  • Logs after initial activity contained only periodic phase=heartbeat events
  • homeboy runner status homeboy-lab reported daemon_freshness.fresh=false, stale_reason_code=binary_hash_mismatch, restartable=false, 33 draining generations, and this job as the final active pin
  • New Agent Task Cook admission failed before provider execution with daemon lease is not fresh

Reviewer reproduction/inspection commands:

homeboy runs show blocks-engine-663-standalone-demo-94c26a4d
homeboy runs evidence blocks-engine-663-standalone-demo-94c26a4d
homeboy runner job logs homeboy-lab d911d948-062f-4ffb-b089-1671290d2177
homeboy runner status homeboy-lab

Expected contract

  • Runner liveness should distinguish wrapper heartbeat from child workload progress.
  • A heartbeat-only workload with no meaningful process or output progress should transition to a diagnosable stalled state under bounded policy.
  • Stalled jobs should expose deterministic cancellation/recovery guidance and should not pin daemon generation promotion forever.
  • Active, genuinely progressing jobs must remain protected during daemon rotation.

Impact

This blocked Homeboy issue #8395 itself from entering provider execution and required manual operator diagnosis. The failure is circular: the starvation fix cannot be cooked on the runner because a heartbeat-only job is currently starving runtime promotion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions