Intra-epoch fast circuit breaker with cooldown + probe traffic recovery

[mingles-agent]

Context

The SPRT-based deactivation (getInactiveStatus) is the existing circuit breaker for offline/degraded nodes. It's accurate but slow — it requires statistically significant evidence before removing a node from the selection pool. During the collection window (10–50+ inferences), a degraded node continues receiving equal traffic.

Live data: nodes reaching 28% miss rates while still in the selection pool. The gap is intra-epoch degradation (node goes offline mid-epoch) that SPRT hasn't caught yet.

Goal

Add a fast, threshold-based circuit breaker inside createFilterFn that:

1. Excludes nodes with high current-epoch miss rates quickly (before SPRT kicks in)
2. Recovers them automatically via cooldown + probe traffic — so a node that fixed a temporary outage can re-enter the pool mid-epoch without waiting for the next epoch boundary

Recovery Design: Cooldown + Probe Traffic

Problem with simple cumulative exclusion: MissedRequests only accumulates within an epoch. A node excluded by miss rate threshold has no way to prove it recovered — it can't receive inferences to rebuild a good track record.

Solution — 3 states for circuit breaker:

HEALTHY  →(miss rate > 25%, ≥4 samples)→  EXCLUDED
EXCLUDED →(cooldown X blocks passed)→      PROBE
PROBE    →(next inference OK)→             HEALTHY
PROBE    →(next inference miss)→           EXCLUDED (cooldown doubles, exponential backoff)

State tracked in a new keeper map: CircuitBreakerState { address, excludedAtBlock, cooldownBlocks, probeAttempts }

Cooldown parameters (configurable via ValidationParams):

• HealthCBInitialCooldownBlocks = 50 blocks (~5 min)
• HealthCBMaxCooldownBlocks = 500 blocks (~50 min)
• Backoff: each re-exclusion doubles cooldown, capped at max

PROBE state mechanics:

• Node in PROBE gets 1 "test slot" in createHealthFilterFn: included with weight=1 regardless of miss rate
• If it completes the inference → state → HEALTHY, cooldown reset
• If it misses → state → EXCLUDED with doubled cooldown
• Probe happens automatically when cooldown expires — no operator action needed

Epoch boundary: On new epoch, all CircuitBreakerState entries are cleared (full reset). Epoch-based recovery is the final backstop.

Implementation

1. New keeper state

// inference-chain/x/inference/keeper/circuit_breaker.go
type CBState int
const (
    CBStateHealthy  CBState = 0
    CBStateExcluded CBState = 1
    CBStateProbe    CBState = 2
)

type CircuitBreakerEntry struct {
    Address           string
    State             CBState
    ExcludedAtBlock   int64
    CooldownBlocks    int64
    ProbeAttempts     int32
}

Store in collections.Map[string, CircuitBreakerEntry] keyed by address. Cleared on epoch transition in moveUpcomingToEffectiveGroup.

2. Health filter function

func (k Keeper) createHealthFilterFn(goCtx context.Context, blockHeight int64) func([]*group.GroupMember) []*group.GroupMember {
    // ... load params, thresholds ...
    return func(members []*group.GroupMember) []*group.GroupMember {
        filtered := make([]*group.GroupMember, 0, len(members))
        for _, m := range members {
            cb := k.GetCBEntry(goCtx, m.Member.Address)
            switch cb.State {
            case CBStateProbe:
                filtered = append(filtered, m) // allow probe slot
            case CBStateExcluded:
                cooldownPassed := blockHeight >= cb.ExcludedAtBlock + cb.CooldownBlocks
                if cooldownPassed {
                    k.SetCBState(goCtx, m.Member.Address, CBStateProbe, blockHeight)
                    filtered = append(filtered, m) // promote to probe
                }
                // else: skip (still in cooldown)
            default: // CBStateHealthy
                total := stats.InferenceCount + stats.MissedRequests
                if total >= minSamples && missRate > threshold {
                    k.SetCBState(goCtx, m.Member.Address, CBStateExcluded, blockHeight)
                    continue // exclude, start cooldown
                }
                filtered = append(filtered, m)
            }
        }
        if len(filtered) == 0 { return members } // safety fallback
        return filtered
    }
}

3. Feedback hook — record probe result

When inference completes or times out, update CB state:

In EndBlock → handleInferenceExpiry (already increments MissedRequests), add:

k.RecordCBResult(ctx, executor.Address, false) // miss → re-exclude or double cooldown

In MsgFinishInference handler, add:

k.RecordCBResult(ctx, executor.Address, true) // success → restore to healthy

4. New ValidationParams fields

uint64 health_cb_miss_threshold_pct = N;      // default: 25 (= 25%)
uint64 health_cb_min_samples = N+1;            // default: 4
uint64 health_cb_initial_cooldown_blocks = N+2; // default: 50
uint64 health_cb_max_cooldown_blocks = N+3;    // default: 500

5. Compose with existing filter in createFilterFn

healthFilter := k.createHealthFilterFn(goCtx, sdkCtx.BlockHeight())
// inference phase: apply health filter
// PoC phase: compose pocFilter(healthFilter(members))

Files to Modify

• inference-chain/x/inference/keeper/query_get_random_executor.go — createFilterFn, new createHealthFilterFn
• inference-chain/x/inference/keeper/circuit_breaker.go — new file: CB state CRUD
• inference-chain/x/inference/module/module.go — clear CB state on epoch transition
• inference-chain/x/inference/keeper/msg_server_finish_inference.go — success feedback
• inference-chain/x/inference/module/module.go EndBlock — miss feedback
• inference-chain/proto/inference/inference/params.proto — 4 new fields
• inference-chain/x/inference/types/params.go — defaults + validation

Acceptance Criteria

• Node with >25% miss rate (≥4 samples) is excluded from GetRandomExecutor results
• Excluded node auto-promotes to PROBE state after initial_cooldown_blocks
• PROBE node receives one inference attempt; success → HEALTHY, miss → EXCLUDED with doubled cooldown
• Cooldown doubles on each re-exclusion, capped at max_cooldown_blocks
• All CB state cleared on epoch boundary
• If ALL candidates excluded/no probes ready, filter is bypassed (no empty pool crash)
• Unit tests: exclusion, probe promotion, probe success, probe fail + backoff, all-degraded fallback, epoch reset

Related

• Issue Research: Node health-aware executor selection (offline detection, miss rate, latency) #3 (research)
• Issue Reputation-adjusted executor selection weight at epoch start #4 (reputation-based weight — complementary, different layer)
• inference-chain/x/inference/calculations/status.go — SPRT (existing slow circuit breaker, remains unchanged)

[mingles-agent]

📝 Issue updated at 2026-03-24T18:03:17.977Z

• Title updated
• Description updated
• Reason: User selected Variant 3 (Cooldown + Probe Traffic) for recovery mechanism. Updated design to include full state machine: HEALTHY → EXCLUDED → PROBE → HEALTHY/re-EXCLUDED with exponential backoff.