feat(proxy): add bounded exhaustion retry rounds for model routes

[sozercan]

Summary

Add an opt-in, Vekil-owned retry round after an explicit priority_failover route exhausts every eligible target with adapter-certified, pre-execution throttling responses.

This builds on the ordered model-route executor introduced by #270. The existing one-pass behavior must remain the default.

Motivation

Ordered failover handles an unavailable or throttled primary, but a short quota-window burst can cause every target to reject the same operation. Vekil currently returns the final throttling response and relies on the caller to retry. A single bounded, delayed retry round could transparently absorb brief quota-window boundaries for clients that do not implement retry behavior.

This feature is a burst absorber, not a substitute for deployment capacity, quota increases, or active-active load distribution.

Proposed configuration

routing:
  mode: priority_failover
  max_target_attempts: 3
  max_upstream_sends: 6
  exhaustion_retry:
    max_rounds: 1
    max_wait_ms: 2000

exhaustion_retry is optional and disabled when omitted. Jitter and internal retry-admission limits should remain implementation-owned rather than expanding the public configuration surface.

Eligibility contract

A new round may begin only when every attempt in the exhausted round satisfies all of the following:

• adapter-certified, authoritative pre-execution 429 or overload rejection;
• definitely replay-safe delivery;
• no semantic output or tool activity;
• no downstream HTTP/SSE/WebSocket commitment;
• no cross-target provider-state binding;
• request, response body, goroutines, and cleanup fully terminated;
• inbound client remains connected;
• shutdown has not started;
• the operation deadline can accommodate the wait and another attempt;
• the request-global upstream-send budget has capacity;
• the route retry-admission limit has capacity.

Any ambiguous transport outcome, malformed response, authentication/configuration error, partial output, or lifecycle uncertainty must suppress the additional round.

Retry behavior

• Permit at most one additional round initially.
• Preserve the original immutable logical request and prepare fresh target-specific requests for every send.
• Keep the global commitment, state-binding, send count, deadline, and failure-precedence state monotonic across rounds.
• Do not reset the request-global upstream-send budget.
• Wait for a bounded provider-directed delay, apply randomized jitter, and make the wait context-aware.
• Re-check cancellation, shutdown, commitment, state binding, and budgets at the second-round dispatch linearization point.
• If the second round exhausts, return the final throttling response with an accurate remaining Retry-After value.

State-bound requests

Cross-target retry rounds are out of scope for provider-state-bound requests. A future same-target delayed retry may be considered separately when an authoritative rejection proves the provider did not execute the request.

Internal design

Model rounds explicitly in the deep route-executor module. Do not implement this as a handler-level loop or by clearing attemptedTargets ad hoc.

Suggested internal state:

type routeOperation struct {
    round              int
    maxRounds          int
    attemptedThisRound map[string]struct{}

    // Monotonic across all rounds:
    remainingUpstreamSends int
    commitment             downstreamCommitment
    hardPinned             bool
    upstreamSends          int
}

The existing handler interface should remain unchanged.

Retry amplification controls

• Bound concurrent waiting retry rounds per route.
• Add randomized jitter to avoid synchronized wakeups.
• Never exceed the configured request-global send budget.
• Emit a suppression reason when retry admission is full.
• Do not automatically loop under sustained route exhaustion.

Observability

Add bounded fields/counters for:

• route round on each physical attempt;
• exhaustion waits;
• exhaustion retry admissions;
• exhaustion retry successes;
• exhaustion retry suppressions by closed reason;
• wait duration;
• final exhaustion after all allowed rounds.

route_exhaustions should count only final exhaustion, not the end of an intermediate round.

Required tests

• all targets return certified 429; delayed retry starts at the first target and succeeds;
• all targets return certified 429 in both rounds; final 429 is returned;
• differing, missing, malformed, and excessive Retry-After values;
• jittered concurrent waiters do not wake simultaneously;
• retry-admission limit suppresses excess loops;
• client cancellation during the wait;
• shutdown during the wait;
• operation deadline expires during the wait;
• cancellation races the second-round dispatch linearization point;
• one ambiguous delivery suppresses the new round;
• partial text/reasoning/tool output suppresses the new round;
• state-bound operations do not change targets;
• exact maximum target-attempt and upstream-send counts;
• response body and goroutine cleanup before round transition;
• client-request and physical-attempt accounting remain exact;
• deterministic fake-clock and full race coverage;
• bounded live burst smoke after deterministic tests pass.

Non-goals

• unlimited retry loops;
• active-active or weighted balancing;
• automatic cross-target state migration;
• cross-model or cross-route fallback;
• hiding sustained capacity shortages;
• generic circuit-breaker or health-probe frameworks.

Acceptance criteria

• Existing configurations retain one-pass behavior without observable changes.
• The feature is explicitly opt-in and schema validation rejects unsafe budgets.
• No request exceeds its target-attempt, round, send, or deadline budget.
• No postcommit, ambiguous-delivery, or state-binding violation can begin another round.
• Retry amplification and concurrent waiters remain bounded.
• All deterministic, lifecycle, race, benchmark, and live burst gates pass.