Research: Testing plan for #4 (reputation-adjusted weight) and #5 (circuit breaker + probe recovery)

[mingles-agent]

Background

Goal

Design a comprehensive, layered test plan for issues #4 and #5 before they go to implementation. The plan must give confidence that the changes work correctly in isolation AND in the live network scenario.

Changes Being Tested

Issue #4 — Reputation-adjusted selection weight

• addEpochMembers in module.go: selection weight = stakeWeight * reputation / 100 (floor 1%)
• New helper calculateSelectionWeight
• Affects: which nodes get how much executor traffic each epoch

Issue #5 — Intra-epoch circuit breaker + cooldown/probe recovery

• New state machine: HEALTHY → EXCLUDED → PROBE → HEALTHY/re-EXCLUDED
• createHealthFilterFn in query_get_random_executor.go
• New keeper: CircuitBreakerState per-participant, cleared on epoch boundary
• Feedback hooks in MsgFinishInference and EndBlock/handleInferenceExpiry
• New ValidationParams: miss threshold (25%), min samples (4), initial cooldown (50 blocks), max cooldown (500 blocks)

Test Infrastructure Available

Unit tests (Go): inference-chain/x/inference/keeper/*_test.go

• Pattern: keepertest.InferenceKeeper(t) gives keeper + context
• Mock-based, fast, no chain needed
• Example: participant_test.go, participant_status_test.go

Integration/E2E tests (Kotlin/Testermint): testermint/src/test/kotlin/

• Full cluster spun up via initCluster(reboot = true)
• Real epoch transitions, real inference flows
• Relevant existing tests: InferenceTests.kt, InferenceFailureAccountingTests.kt, ValidationTests.kt, MultiNodeTests.kt, ParticipantPowerTests.kt
• Test isolation via genesis spec overrides (e.g. ValidationParams can be set in genesis for test params)

What to Investigate

1. Unit test scenarios for #4 (reputation weight)

Map out test cases for calculateSelectionWeight and addEpochMembers:

• reputation=100 → weight = stakeWeight (no change)
• reputation=50 → weight = stakeWeight * 0.5
• reputation=0 → weight = floor (1% of stakeWeight)
• reputation=1 → weight = floor or 1% (boundary)
• Multiple participants with different reputations: verify relative selection probability distribution
• Does existing GetRandomMemberForModel test cover weighted selection? Check inference-chain/x/inference/keeper/ for existing random executor tests

2. Unit test scenarios for #5 (circuit breaker)

Map out all state machine transitions:

• HEALTHY → EXCLUDED: miss rate > 25%, ≥4 samples triggers exclusion
• HEALTHY: miss rate > 25% but <4 samples → NOT excluded (insufficient data)
• HEALTHY: miss rate ≤ 25% with any sample count → stays HEALTHY
• EXCLUDED: cooldown not passed → still excluded from filter
• EXCLUDED: cooldown passed → promoted to PROBE
• PROBE → HEALTHY: successful inference recorded
• PROBE → EXCLUDED: miss → doubles cooldown
• PROBE → EXCLUDED (3rd time): cooldown = initial × 4, capped at max
• All-degraded fallback: all nodes excluded → filter returns original list (no empty pool)
• Epoch boundary: all CB state cleared regardless of current state

3. Integration test scenarios (Testermint)

Examine existing tests to understand how to set up controlled miss scenarios:

• How does InferenceFailureAccountingTests.kt simulate node failures?
• How does ValidationTests.kt verify participant status changes?
• How are ValidationParams overridden in test genesis specs?

Propose Testermint test scenarios:

• Scenario A (reputation weight): Two nodes, different reputation histories (manipulate EpochPerformanceSummary). Verify over 100 inference requests that high-reputation node receives ~2× more requests than 50-reputation node
• Scenario B (fast exclusion): One node starts missing inferences (simulate via node shutdown or mock). Verify it stops receiving requests within 4 misses, not after SPRT threshold
• Scenario C (probe recovery): Excluded node's cooldown expires → verify it receives exactly one probe inference → bring node back up → verify HEALTHY state and normal traffic resumes
• Scenario D (exponential backoff): Node keeps failing probes → verify cooldown doubles each time up to max
• Scenario E (network resilience): All nodes briefly offline → verify filter fallback prevents empty pool, traffic resumes when any node recovers

4. Param testing

How to test with non-default params (e.g., lower min_samples=2 for faster test execution):

• Identify if ValidationParams can be overridden in Testermint genesis spec (check SpecTests.kt and TestUtils.kt)
• What's the minimum epoch length for integration tests? (affects cooldown block counts — may need to set health_cb_initial_cooldown_blocks very low, e.g. 5 blocks for tests)

5. Regression risks

What existing behavior could break:

• GetRandomExecutor returning empty result (most critical — network stops)
• Nodes incorrectly excluded due to param misconfiguration
• CB state not cleared on epoch boundary (nodes permanently excluded)
• Reputation calculation regression affecting existing nodes' weights
• Interaction between Reputation-adjusted executor selection weight at epoch start #4 weight changes and SPRT (SPRT removes nodes entirely, weight system just reduces probability — ensure they don't conflict)

Check: are there existing tests for GetRandomExecutor that need updating?

Key Files

• inference-chain/x/inference/keeper/participant_test.go — unit test patterns
• inference-chain/x/inference/keeper/participant_status_test.go — SPRT test patterns
• testermint/src/test/kotlin/InferenceFailureAccountingTests.kt — failure simulation
• testermint/src/test/kotlin/ValidationTests.kt — participant status E2E
• testermint/src/test/kotlin/MultiNodeTests.kt — multi-node scenarios
• testermint/src/test/kotlin/ParticipantPowerTests.kt — power/weight tests
• testermint/src/test/kotlin/TestUtils.kt — test helpers

Deliverables

1. Unit test specification: exact test function names + inputs/assertions for all cases in Reputation-adjusted executor selection weight at epoch start #4 and Intra-epoch fast circuit breaker with cooldown + probe traffic recovery #5
2. Testermint integration test specification: test class names, setup, assertions, required genesis param overrides
3. Regression test checklist: which existing tests to run + any that need updating
4. Test execution order: what to run in CI vs manual validation
5. Create implementation task: "Write tests for Reputation-adjusted executor selection weight at epoch start #4 and Intra-epoch fast circuit breaker with cooldown + probe traffic recovery #5 per test plan"

Focus Areas

• unit test patterns in keeper_test
• testermint integration test setup
• controlled miss simulation
• genesis param overrides for test
• regression risks

[mingles-agent]

🏗️ ARCHITECT: ## Research Findings — Testing Plan for Issues #4 and #5

---

Problem Statement

Before issues #4 and #5 are merged, we need a layered test plan covering: (a) unit correctness of all new logic, (b) integration proof that the components work together in a real chain, and (c) no regressions to existing executor selection and participant accounting.

---

Current State

Substantial implementation already exists. Audit of the codebase found:

Issue #4 — Reputation-adjusted weight (epochgroup/)

• CalculateSelectionWeight(stakeWeight, reputation int64) int64 — implemented in epochgroup/epoch_group.go:60
• buildSelectionWeightsMap() — implemented in epochgroup/random.go:78
• addEpochMembers — implemented in module/module.go:747, calls calculateParticipantReputation → NewEpochMemberFromActiveParticipant → upcomingEg.AddMember
• Unit tests already exist in epochgroup/epoch_group_test.go:
  • TestCalculateSelectionWeight_* — 8 cases covering rep=100, rep=0 (floor), negative stake, 50%, boundary
  • TestSelectionWeights_ProportionalTraffic — 100k-draw Monte Carlo verifying 2:1 ratio
  • TestSelectionWeights_ZeroReputationGetsFloor — floor enforcement
  • TestBuildSelectionWeightsMap — map construction
  • TestValidationWeightsUnchanged — immutability of PoC voting weights

Issue #5 — Circuit breaker (keeper/)

• State machine + persistence: circuit_breaker.go — fully implemented
• createHealthFilterFn — implemented in query_get_random_executor.go:128
• Feedback hooks:
  • Success: msg_server_finish_inference.go:308 calls RecordCBResult(..., true) via handleInferenceCompleted
  • Miss/expiry: module.go:327 calls RecordCBResult(..., false) after incrementing MissedRequests
  • Epoch boundary clear: module.go:908 calls ClearAllCBState
• Unit tests already exist in circuit_breaker_test.go:
  • TestCBStateDefaultsToHealthy, TestCBSetAndGet — persistence
  • TestExcludeCBEntry, TestExcludeDoublesCooldownOnReExclusion, TestExcludeCapsCooldownAtMax — exclusion
  • TestPromoteCBEntryToProbe, TestRecordCBResult_ProbeSuccess, TestRecordCBResult_ProbeFailure, TestRecordCBResult_HealthyNodeNoOp — state machine
  • TestClearAllCBState — epoch boundary
  • TestHealthFilter* (7 tests) — filter function covering all CB states, cooldown expiry, below-min-samples, and all-degraded fallback

---

Gaps Identified

Despite solid unit coverage, the following are missing:

A. Missing unit tests

#4 — Reputation weight:

1. TestAddEpochMembers_ReputationFlowsToValidationWeights — verify that addEpochMembers stores Reputation in EpochGroupData.ValidationWeights (not just passed through silently). Currently addEpochMembers is called only from the AppModule level and has no keeper-level unit test.
2. TestCalculateSelectionWeight_Boundary_ExactThreshold — reputation=1 → adjusted=stakeWeight/100, floor=stakeWeight/100, they're equal; test the exact equality edge.
3. TestHealthFilter_ExactMissThreshold — missedRequests*100 == DefaultCBMissThresholdPct*total should NOT trigger exclusion (strict > not >=). Currently untested boundary.

#5 — Circuit breaker feedback hooks:
4. TestFinishInference_RecordsCBSuccess — unit test for MsgFinishInference that verifies RecordCBResult(success=true) transitions PROBE → HEALTHY. Use keeper test pattern from participant_test.go.
5. TestExpiredInference_RecordsCBMiss — unit test for the inference expiry path in module.go:handleExpiredInference that verifies MissedRequests++ AND RecordCBResult(false). This tests the wiring, not just the CB functions in isolation.
6. TestHealthFilter_MissRateCalculation_InferenceCountVsMissedRequests — the current filter uses total = InferenceCount + MissedRequests. Verify this is consistent with how the counts are incremented (success increments InferenceCount, expiry increments MissedRequests — both contribute to total).

B. Critical design finding: CB params NOT in proto yet

DefaultCBMissThresholdPct, DefaultCBMinSamples, DefaultCBInitialCooldownBlocks, DefaultCBMaxCooldownBlocks are Go constants (circuit_breaker.go:31-44), not proto ValidationParams fields.

This has two implications:

• Integration tests cannot override these via genesis spec (unlike SPRT params which are in ValidationParams)
• The Testermint integration test scenarios requiring fast cooldowns (e.g. 5-block cooldown for test speed) are not possible without promoting these to proto

Recommendation: Promote CB params to ValidationParams proto as part of the implementation (or as a sub-task). Until then, integration test scenarios B/C/D require ~50 block waits.

C. Missing Testermint integration tests

No Testermint tests cover #4 or #5. The getCBState query endpoint doesn't exist in the client yet either. The following are missing:

1. ReputationWeightDistributionTest — verify that nodes with different EpochsCompleted (reputation) receive proportionally different executor traffic
2. CircuitBreakerFastExclusionTest — verify a failing node stops receiving traffic within min_samples inferences
3. CircuitBreakerProbeRecoveryTest — full EXCLUDED → PROBE → HEALTHY cycle
4. CircuitBreakerExponentialBackoffTest — verify cooldown doubles on repeated probe failures
5. CircuitBreakerAllDegradedFallbackTest — verify network doesn't crash when all nodes are excluded

---

Detailed Unit Test Specification

All tests go in the appropriate *_test.go file in inference-chain/x/inference/keeper/ or epochgroup/.

File: epochgroup/epoch_group_test.go (additions)

// TestCalculateSelectionWeight_ExactFloorEquality verifies when adjusted == floor
func TestCalculateSelectionWeight_ExactFloorEquality(t *testing.T) {
    // stake=1000, rep=1: adjusted = 1000*1/100 = 10, floor = 1000/100 = 10 → return 10
    require.Equal(t, int64(10), CalculateSelectionWeight(1000, 1))
}

// TestCalculateSelectionWeight_3NodeDistribution verifies 3 nodes with reps
// 100/50/0 receive approximately 1000:500:10 traffic shares
func TestCalculateSelectionWeight_3NodeDistribution(t *testing.T) {
    // Run 200k draws; tolerance ±2%
    // expected: high=1000/(1510)≈66.2%, mid=500/1510≈33.1%, low=10/1510≈0.66%
}

File: keeper/circuit_breaker_test.go (additions)

// TestHealthFilter_ExactMissThreshold verifies strict > (not >=) threshold
func TestHealthFilter_ExactMissThreshold(t *testing.T) {
    // 1 hit, 3 misses → total=4 ≥ min_samples, missedRequests*100 = 300, threshold*total = 25*4=100
    // 300 > 100 → EXCLUDED (57% > 25%)
    // 3 hits, 1 miss → total=4, missedRequests*100=100, threshold*total=100 → NOT > → included
    // Tests boundary: exactly 25% miss rate should NOT exclude (strictly >)
}

// TestHealthFilter_MissCountVsInferenceCount verifies the total calculation
func TestHealthFilter_InferenceCountAndMissedRequestsSumToTotal(t *testing.T) {
    // InferenceCount=3, MissedRequests=1 → total=4, miss pct=25% → included
    // InferenceCount=2, MissedRequests=2 → total=4, miss pct=50% → excluded
}

File: keeper/msg_server_finish_inference_test.go (additions)

// TestFinishInference_ProbeCBTransitionsToHealthy
// Setup: put node in PROBE state, submit MsgFinishInference, verify CB entry deleted
func TestFinishInference_ProbeCBTransitionsToHealthy(t *testing.T) {
    k, ctx, mocks := keepertest.InferenceKeeperReturningMocks(t)
    // ... set executor to PROBE
    // ... call k.FinishInference(ctx, msg)
    // ... verify k.GetCBEntry(ctx, executorAddr).State == CBStateHealthy
}

File: keeper/endblock_inference_expiry_test.go (new file or additions)

// TestExpiredInference_MissedRequestsAndCBResult
// Verify that when an inference expires: MissedRequests increments AND
// RecordCBResult(false) is called (which transitions PROBE→EXCLUDED or no-ops on HEALTHY)
func TestExpiredInference_UpdatesMissedRequestsAndCB(t *testing.T) { ... }

---

Detailed Testermint Integration Test Specification

Prerequisites / blockers

1. CB params must be in ValidationParams proto — without this, cooldown tests need 50-block waits (possible but slow)
2. Add getCBState(address) API query to Testermint client — needed by scenarios B/C/D to verify CB transitions

Alternative for immediate testing without proto promotion: Use a very short epoch (~60 blocks) and accept ~50-block cooldown waits within a single test. Feasible but slow.

File: testermint/src/test/kotlin/CircuitBreakerTests.kt (new)

@Timeout(value = 20, unit = TimeUnit.MINUTES)
class CircuitBreakerTests : TestermintTest() {

    val cbTestSpec = spec {
        this[AppState::inference] = spec<InferenceState> {
            this[InferenceState::params] = spec<InferenceParams> {
                this[InferenceParams::validationParams] = spec<ValidationParams> {
                    // If CB params are promoted to proto:
                    // this[ValidationParams::healthCbInitialCooldownBlocks] = 5L
                    // this[ValidationParams::healthCbMinSamples] = 2L
                }
            }
        }
    }

    @Test
    fun `fast exclusion - node stops receiving traffic after min_samples misses`() {
        // 1. initCluster with 2 nodes
        // 2. Mock one node to fail all inferences (setInferenceResponse("bad json"))
        // 3. Run inferences until the failing node accumulates DefaultCBMinSamples(4) misses
        // 4. Run 20 more inferences — verify failing node receives 0 (CB excluded)
        // 5. Check CB state via query
    }

    @Test
    fun `probe recovery - excluded node recovers after cooldown`() {
        // 1. initCluster with 2 nodes
        // 2. Mock join1 to fail → get excluded
        // 3. Wait DefaultCBInitialCooldownBlocks (50 blocks)
        // 4. Run 1 inference — join1 should be selected as probe
        // 5. Restore join1 mock to succeed
        // 6. Verify join1 transitions to HEALTHY, receives normal traffic next epoch
    }

    @Test
    fun `exponential backoff - cooldown doubles on repeated probe failures`() {
        // 1. Exclude node, wait cooldown → PROBE → fail probe → re-excluded with 2x cooldown
        // 2. Wait 2x cooldown → PROBE → fail again → 4x cooldown
        // 3. Verify final cooldown == DefaultCBMaxCooldownBlocks (capped at 500)
    }

    @Test
    fun `all-degraded fallback - network keeps running when all nodes excluded`() {
        // 1. initCluster with 2 nodes (genesis + join1)
        // 2. Mock ALL nodes to fail inferences
        // 3. Run until both are excluded (requires > DefaultCBMinSamples failures)
        // 4. Verify GetRandomExecutor still returns a node (fallback to full list)
        // 5. Verify inferences are still accepted (not erroring with empty pool)
    }
}

File: testermint/src/test/kotlin/ReputationWeightTests.kt (new)

@Timeout(value = 20, unit = TimeUnit.MINUTES)
class ReputationWeightTests : TestermintTest() {

    @Test
    fun `high reputation node receives more executor traffic than low reputation node`() {
        // 1. initCluster with 2 nodes (genesis + join1)
        // 2. Wait for both to complete several epochs (build up EpochsCompleted)
        //    OR manipulate EpochsCompleted via governance param update
        // 3. Find a participant with ~100 reputation and one with ~50
        //    (or set up fresh cluster where one node participates twice as many epochs)
        // 4. Run 200 inference requests (enough for statistical significance)
        // 5. Count how many went to each node
        // 6. Assert high-reputation node received ≥ 1.5x more than low-reputation node (±15%)
        // Caveat: this test is probabilistic — assert ratio within tolerance, not exact
    }

    @Test
    fun `reputation 0 node still receives floor traffic`() {
        // 1. Start a fresh participant with 0 EpochsCompleted (reputation=0)
        // 2. Run 1000 inferences against a node with full reputation
        // 3. Verify new node received at least 0.5% of traffic (floor applied)
    }
}

---

Regression Test Checklist

Run these existing tests with the #4/#5 branches to catch regressions:

Must-pass (existing tests, no changes needed)

Test	Location	What it guards
All TestCalculateSelectionWeight_*	epochgroup/epoch_group_test.go	Weight formula correctness
TestSelectionWeights_ProportionalTraffic	epochgroup/epoch_group_test.go	Distribution not broken
TestValidationWeightsUnchanged	epochgroup/epoch_group_test.go	PoC weights not mutated
All TestCBState* / TestExclude* / TestRecord*	circuit_breaker_test.go	CB state machine
All TestHealthFilter*	circuit_breaker_test.go	Filter correctness
TestClearAllCBState	circuit_breaker_test.go	Epoch boundary clear
verify failed inference is refunded	InferenceFailureAccountingTests.kt	Refund accounting not broken by CB
All ValidationTests	ValidationTests.kt	SPRT + validation pipeline
reputation increases after epoch participation	ParticipantTests.kt	Reputation calculation unchanged

Tests likely needing updates

Test	Reason
msg_server_start_inference_test.go	If inference state machine changes
Any test counting exact InferenceCount	CB now reads this field — verify mock data is consistent

Regression risks to explicitly check

1. Empty pool crash: GetRandomExecutor returning error when all nodes in CB excluded. Covered by TestHealthFilterFallbackAllDegraded + Scenario E.
2. CB state survives epoch boundary: ClearAllCBState is called at module.go:908 — verify it's inside the onSetNewValidatorsStage path and not accidentally in a conditional.
3. SPRT + CB interaction: SPRT sets ParticipantStatus_INACTIVE/INVALID which removes the node from the epoch group entirely. CB operates on nodes already in the epoch group. These are orthogonal — no conflict expected, but verify that an INACTIVE node is not in the epoch group members list passed to the CB filter.
4. Double-counting total: total = InferenceCount + MissedRequests. Verify InferenceCount increments on handleInferenceCompleted (success) and MissedRequests increments on expiry. Both contribute to total, so 4 completed + 0 missed = total 4 samples (healthy). This is correct but needs explicit test.
5. Nil participant data guard: In createHealthFilterFn, if GetParticipant returns !found, the node is included by default. This is the correct safe fallback — guard this with a test.

---

Test Execution Order (CI vs Manual)

CI (every PR, fast)

1. go test ./x/inference/keeper/... -run TestCB — CB unit tests (~1s)
2. go test ./x/inference/epochgroup/... -run TestCalculateSelectionWeight — weight unit tests (~1s)
3. go test ./x/inference/epochgroup/... -run TestSelectionWeights — distribution tests (100k draws, ~2s)
4. go test ./x/inference/keeper/... -run TestHealthFilter — filter tests (~1s)
5. go test ./x/inference/... — full keeper suite including new tests

Manual / staging (before merge, slower)

1. ReputationWeightTests in Testermint — run on staging cluster
2. CircuitBreakerTests.fast exclusion — needs failing mock setup
3. CircuitBreakerTests.probe recovery — needs 50-block wait
4. CircuitBreakerTests.all-degraded fallback — brief all-node failure scenario
5. Run existing ValidationTests + InferenceFailureAccountingTests to verify no regressions

---

Summary of What Needs to be Written

The implementation task should cover:

New unit tests (Go, ~1–2 days):

• TestCalculateSelectionWeight_ExactFloorEquality (epochgroup)
• TestCalculateSelectionWeight_3NodeDistribution (epochgroup)
• TestHealthFilter_ExactMissThreshold (keeper)
• TestHealthFilter_InferenceCountAndMissedRequestsSumToTotal (keeper)
• TestHealthFilter_NilParticipantIncludeByDefault (keeper)
• TestFinishInference_ProbeCBTransitionsToHealthy (keeper)
• TestExpiredInference_UpdatesMissedRequestsAndCB (keeper)

Optional but recommended (Go, ~0.5 days):

• Promote CB constants to ValidationParams proto (or document that they're hardcoded for v1)

Testermint integration tests (~2 days if CB params in proto, ~3 days if hardcoded):

• CircuitBreakerTests.kt (4 scenarios)
• ReputationWeightTests.kt (2 scenarios)
• Add getCBState(address) API query to Testermint client