Skip to content

IMPLEMENTATION_SUMMARY_1275.md

Latest commit

 

History

History
138 lines (105 loc) · 5.23 KB

IMPLEMENTATION_SUMMARY_1275.md

File metadata and controls

138 lines (105 loc) · 5.23 KB
Raw

Fix for Issue #1275: Cache Consistency Across Replicas

Summary of Changes

This PR addresses cache consistency issues across event-indexer replicas by:

  1. Adding TTL support to EventCache to bound staleness
  2. Adding leadership-loss cache invalidation
  3. Exposing ApiCache backend status in the health endpoint
  4. Adding comprehensive adversarial tests
  5. Updating documentation

Changes Made

1. EventCache TTL Support (services/event-indexer/src/cache.rs)

Added:

  • CachedEntry struct wrapping events with insertion timestamps
  • ttl_secs field to EventCache (default: 5 minutes)
  • with_ttl() constructor for custom TTL configuration
  • is_expired() helper method checking entry age
  • TTL filtering in get() and get_by_match() methods

Effect:

  • Former leaders can no longer serve indefinitely-stale data
  • Staleness is bounded to TTL window (5 minutes default)
  • After TTL, cache misses force DB fallback (authoritative source)

Tests Added:

  • ttl_causes_cache_miss_after_expiry
  • get_by_match_filters_expired_entries
  • reinsertion_refreshes_ttl

2. Leadership-Loss Cache Invalidation (services/event-indexer/src/rpc.rs)

Modified event_poller():

  • Added was_leader state tracking
  • Detects leadership loss transition
  • Calls cache.clear() immediately on leadership loss
  • Logs warning when clearing cache

Effect:

  • Eager invalidation on known leadership changes
  • Complements TTL for immediate consistency on failover
  • Prevents stale reads from former leaders

3. Health Endpoint Cache Backend Reporting (services/event-indexer/src/api.rs)

Modified HealthResponse:

  • Added cache_backend: String field
  • Added cache_shared: bool field

Modified health_check():

  • Reports cache_backend (from ApiCache::backend_name())
  • Reports cache_shared (from ApiCache::is_shared())
  • Returns 503 degraded when cache_shared is false

Effect:

  • External monitoring can detect non-shared cache degradation
  • No longer a silent failure logged only to STDERR
  • Load balancers can route away from degraded instances

4. Improved Warning Message (services/event-indexer/src/api_cache.rs)

Updated ApiCache::from_config():

  • Corrected warning from "latency will be higher" to describe actual consistency risk
  • Now explicitly states: "multiple replicas will serve different cached responses"
  • References health endpoint reporting

Effect:

  • Operators understand this is a correctness issue, not just performance
  • Clear guidance on detection mechanism

5. Comprehensive Adversarial Tests (services/event-indexer/tests/cache_consistency_tests.rs)

New test file covering:

  • Split-brain scenario: two replicas with different cached state
  • Convergence after TTL expiry
  • Former leader staleness bounded by TTL
  • ApiCache backend detection via is_shared() and backend_name()
  • Leadership loss cache clearing

Test scenarios:

  • two_replicas_different_events_diverge_before_ttl - proves the bug exists before TTL
  • two_replicas_converge_after_ttl - proves TTL bounds staleness
  • former_leader_serves_stale_data_within_ttl_window - proves bounded staleness window
  • api_cache_redis_unreachable_detectable - proves health endpoint detection
  • leadership_loss_clears_cache - proves eager invalidation

6. Updated Health Check Tests (services/event-indexer/tests/health_check_tests.rs)

Added tests:

  • health_check_reports_cache_backend - verifies cache_backend field
  • health_check_reports_cache_shared - verifies cache_shared field
  • health_check_degraded_when_cache_not_shared - verifies degraded status
  • health_check_disabled_cache_not_shared - verifies disabled cache behavior

7. Documentation Updates (docs/event-indexer-scaling.md)

Added sections:

  • TTL and staleness bounds - explains TTL mechanism and bounded staleness
  • Leadership-loss invalidation - documents eager cache clearing
  • Multi-replica consistency - documents consistency guarantees
  • Redis unavailable (ApiCache degradation) - failure mode documentation

Updated:

  • LRU Cache section with TTL details
  • Failure Modes section with Redis fallback monitoring

Verification Checklist

  • TTL mechanism added to EventCache
  • Leadership-loss cache clearing implemented
  • Health endpoint exposes cache backend status
  • ApiCache warning message corrected
  • Adversarial tests cover split-brain scenarios
  • Health check tests cover new fields
  • Documentation updated with consistency guarantees
  • All existing tests should still pass (LRU semantics unchanged)
  • DB-authoritative fallback path preserved

Consistency Guarantee

After this fix:

Before: Former leader serves stale cache indefinitely → unbounded staleness

After: Staleness bounded to max(TTL_SECS, time_since_detected_leadership_loss)

  • Leadership loss detected → immediate cache clear
  • Undetected failure / split-brain → TTL bounds staleness to 5 minutes
  • All replicas converge to DB state after window

Migration Notes

  • EventCache default TTL: 5 minutes (300 seconds)
  • Can be overridden via EventCache::with_ttl(size, ttl_secs)
  • No breaking API changes
  • Existing cache users automatically get TTL behavior
  • Health endpoint response schema expanded (backwards compatible JSON)