bug: internal LLM calls have no timeout — tasks hang indefinitely in batch runs

[ZuyiZhou]

Symptom

Agent-internal LLM API calls (main loop and sub-agents) are issued without a timeout. When the backend or an intermediate gateway stalls on a single request, the whole task hangs forever: process alive, zero new messages, no error — only an external watchdog eventually kills it.

In one 1,000+-task batch rollout, roughly 1 in 6 tasks entered this state. These are not long-but-legitimate runs (the legitimate long tail in our workloads is ~30 min with steady tool activity); hung tasks produce no output at all after launch.

Current workaround

We run a local patch that injects a per-call timeout via an env-var override, plus an external re-roll pass for tasks that failed that way. It works, but it's a fork we'd like to drop.

Proposal

Make this first-class configuration:

• llm_call_timeout — per call, applying to main-loop and sub-agent calls alike
• a retry budget (max retries per call and/or per run) with backoff
• graceful finalization: when the budget is exhausted, the run terminates with a structured error result (see feat(observability): structured termination reason for every run #149) instead of hanging or dying silently

Happy to upstream our patch as a starting point if the direction sounds right.