Skip to content

FAILURE_MODES.md

Latest commit

 

History

History
73 lines (64 loc) · 5.14 KB

FAILURE_MODES.md

File metadata and controls

73 lines (64 loc) · 5.14 KB

Comprehensive Failure Modes Documentation

This document outlines the known failure modes across the TeachLink ecosystem, including smart contracts, indexer infrastructure, and integrations. It provides an assessment of impact, recovery procedures, and strategies for prevention for each identified failure mode.

1. Smart Contract & Protocol Failures

1.1. Unexpected State Reversion or Invariant Violation

  • Description: A smart contract execution reverts unexpectedly due to an invalid state transition or failed invariant check (e.g., math overflow, incorrect permissions).
  • Impact Assessment: High (SEV2). Users cannot complete critical transactions (e.g., payments, escrows, bridges). Depending on the affected function, it could lead to partial feature outages.
  • Recovery Procedures:
    1. Halt affected user flows on the frontend immediately.
    2. Engage the incident response team to triage the specific transaction hash causing the reversion.
    3. Formulate a hotfix or state migration if the contract state is permanently bricked, and upgrade the contract.
    4. Communicate the issue and resolution to affected users.
  • Prevention Strategies:
    • Extensive unit testing, property-based testing, and fuzzing.
    • Formal verification of core invariant logic.
    • Pre-deployment dry runs on testnet environments.

1.2. Extreme Network Fee (Gas) Spikes

  • Description: The underlying network experiences significant congestion, causing transaction fees to exceed predefined thresholds or baseline limits.
  • Impact Assessment: Medium (SEV3). Decreased user experience, delayed transactions, and possible failed transaction submissions if gas limits are improperly configured.
  • Recovery Procedures:
    1. Monitor gas_thresholds.json and adjust the frontend baseline estimation multipliers temporarily.
    2. Implement an automated retry mechanism with dynamic fee bumping (if applicable).
    3. Advise users via UI announcements about network congestion.
  • Prevention Strategies:
    • Dynamic fee estimation integrated into the client application.
    • Efficient contract design to minimize instruction and state size overhead.
    • Implement queuing mechanisms for non-time-sensitive operations during peak spikes.

2. Infrastructure & Backend Failures

2.1. Indexer Database Corruption or Data Loss

  • Description: The PostgreSQL database supporting the indexer experiences data corruption, accidental deletion, or storage failure.
  • Impact Assessment: Critical (SEV1). The frontend will display inaccurate or completely missing off-chain data, and analytical/observability queries will fail.
  • Recovery Procedures:
    1. Escalate to SEV1 per the Incident Response playbook.
    2. Pause the indexer service to prevent further data corruption.
    3. Execute disaster recovery procedures: utilize DISASTER_RECOVERY_PROCEDURES.md to restore from the latest verified backup manifest.
    4. Perform data reconciliation against on-chain state for the delta between the backup and the failure timestamp.
    5. Restart the indexer and verify data integrity.
  • Prevention Strategies:
    • Routine backup integrity verification (SLO: >= 99% success rate).
    • High availability (HA) database configurations with active replication.
    • Regular restoration drills as specified in the DR procedures.

2.2. Horizon Node Desynchronization

  • Description: The RPC or Horizon node endpoint falls out of sync with the network, providing stale data or rejecting valid transaction submissions.
  • Impact Assessment: High (SEV1/SEV2). Both the application backend and indexer will process stale data, leading to failed states and incorrect user balances.
  • Recovery Procedures:
    1. Immediately failover to a secondary or fallback Horizon/RPC endpoint.
    2. Verify network connectivity and DNS routing.
    3. Purge stale cache entries in the indexer and force a re-sync of the affected block range.
  • Prevention Strategies:
    • Utilize a load-balanced pool of RPC nodes across multiple providers.
    • Implement health checks that validate the latest block timestamp against the current UTC time.
    • Automated alerts for event-processing lag or stale blocks.

3. Integration & Third-Party Failures

3.3. Stuck Cross-Chain Bridge Transactions

  • Description: A transaction initiated to bridge assets across chains becomes stuck due to relayer failure, signature validation issues, or target chain congestion.
  • Impact Assessment: High (SEV1). User funds are temporarily inaccessible, creating immediate trust and financial impact.
  • Recovery Procedures:
    1. Identify the stuck transaction ID and affected users.
    2. Determine if the failure occurred on the source chain, relayer network, or target chain.
    3. Execute manual or administrative retry commands via the relayer interface.
    4. If manual retry fails, fallback to manual refund procedures if the protocol allows.
  • Prevention Strategies:
    • Comprehensive cross-chain integration tests (CROSS_CHAIN_INTEGRATION_TESTS.md).
    • Active monitoring and alerting on bridge queue depths and processing times.
    • Implement time-locked escrow refunds for transactions stuck beyond a specific threshold.