Merge pull request #1032 from Favourof/feat/issue-944-951-circuit-breaker-encryption

feat(backend): implement circuit breaker (#944) and data encryption a…

Author: Hydrax117

backend/CIRCUIT_BREAKER.md

@@ -0,0 +1,94 @@
+# External Service Circuit Breaker (Issue #944)
+
+## Overview
+The External Service Circuit Breaker protects the ArenaX backend from cascading failures caused by outages, network timeouts, or degraded performance in third-party services (e.g. Stellar Horizon, Soroban RPC, external payment providers, webhooks).
+
+## States & State Transitions
+
+```mermaid
+stateDiagram-v2
+    [*] --> Closed
+    Closed --> Open: Failures >= threshold
+    Open --> HalfOpen: Cooldown period elapsed
+    HalfOpen --> Closed: Consecutive successes >= success_threshold
+    HalfOpen --> Open: Trial request fails (exponential backoff scaled)
+```
+
+1. **Closed**: Normal state. All requests are allowed through. Consecutive failures are tracked. When failures reach `failure_threshold` (default: 5), the circuit trips to `Open`.
+2. **Open**: Outage state. Incoming calls fail fast immediately with `503 Service Unavailable` and a `Retry-After` header without making any network calls to the external service.
+3. **HalfOpen**: Recovery state. Once the exponential backoff cooldown has elapsed, the circuit admits a limited number of trial probe requests (`half_open_probe_limit`, default: 1).
+   - If the trial request succeeds and reaches `success_threshold` (default: 2), the circuit transitions back to `Closed` and resets the backoff exponent.
+   - If the trial request fails, the circuit immediately returns to `Open` and scales the cooldown duration exponentially.
+
+## Exponential Backoff Calculation
+
+The cooldown period before transitioning from `Open` to `HalfOpen` grows exponentially with consecutive trip cycles:
+
+$$\text{cooldown} = \min\left(\text{max\_backoff}, \; \text{initial\_backoff} \times \text{backoff\_multiplier}^{(\text{trip\_count} - 1)}\right)$$
+
+### Default Settings
+- `failure_threshold`: 5 consecutive failures
+- `success_threshold`: 2 consecutive successes
+- `initial_backoff`: 5 seconds
+- `max_backoff`: 60 seconds
+- `backoff_multiplier`: 2.0
+- `half_open_probe_limit`: 1 concurrent probe
+- `request_timeout`: 10 seconds
+
+## Usage
+
+### 1. Service Registry Wrapper
+```rust
+use arenax_backend::middleware::circuit_breaker::CircuitBreakerRegistry;
+
+let registry = CircuitBreakerRegistry::default();
+
+// Execute outbound calls under the "stellar" circuit breaker
+let result = registry.call("stellar", || async {
+    // Outbound HTTP request to Stellar Horizon
+    reqwest::get("https://horizon.stellar.org/accounts/XYZ").await
+}).await;
+
+match result {
+    Ok(data) => println!("Success: {:?}", data),
+    Err(CircuitBreakerError::CircuitOpen { service, retry_after_secs }) => {
+        eprintln!("Fast failed: {} circuit is open, retry in {}s", service, retry_after_secs);
+    }
+    Err(CircuitBreakerError::Timeout(dur)) => {
+        eprintln!("Call timed out after {:?}", dur);
+    }
+    Err(CircuitBreakerError::Inner(e)) => {
+        eprintln!("External call failed: {:?}", e);
+    }
+}
+```
+
+### 2. Actix-Web Route Middleware
+```rust
+use actix_web::{web, App};
+use arenax_backend::middleware::circuit_breaker::{
+    CircuitBreaker, CircuitBreakerConfig, ExternalCircuitBreakerMiddleware,
+};
+
+let breaker = Arc::new(CircuitBreaker::new(
+    CircuitBreakerConfig::new("proxy_service")
+        .with_failure_threshold(3)
+        .with_initial_backoff(Duration::from_secs(10)),
+));
+
+App::new()
+    .service(
+        web::scope("/api/external")
+            .wrap(ExternalCircuitBreakerMiddleware::new(breaker.clone()))
+            .route("/proxy", web::get().to(proxy_handler))
+    );
+```
+
+## Metrics Export
+The circuit breaker publishes Prometheus metrics via the `/metrics` endpoint:
+
+| Metric Name | Type | Labels | Description |
+|-------------|------|--------|-------------|
+| `circuit_breaker_state` | Gauge | `service` | Current state (0 = Closed, 1 = HalfOpen, 2 = Open) |
+| `circuit_breaker_requests_total` | Counter | `service`, `status` | Total requests (`success`, `failure`, `rejected`) |
+| `circuit_breaker_trips_total` | Counter | `service` | Total times circuit tripped Open |

backend/Cargo.lock

@@ -41,6 +41,8 @@ futures = "0.3"
 futures-util = "0.3"
 hex = "0.4"
 fs2 = "0.4"
+aes-gcm = { version = "0.10", features = ["zeroize"] }
+zeroize = { version = "1.8", features = ["derive"] }
 
 # Security: pin patched versions of transitive deps
 quinn-proto = "0.11.14"

backend/Cargo.toml

@@ -0,0 +1,64 @@
+# Data Encryption at Rest & Key Rotation (Issue #951)
+
+## Overview
+ArenaX backend implements authenticated data encryption at rest for Personally Identifiable Information (PII) and sensitive records (phone numbers, emails, device fingerprints, secrets) using **AES-256-GCM** (NIST SP 800-38D).
+
+## Key Features
+
+1. **Authenticated Encryption (AEAD)**:
+   - Uses AES-256-GCM providing both confidentiality and integrity/tamper detection.
+   - Generates a unique, cryptographically random 96-bit nonce per encryption operation via `OsRng`.
+2. **Versioned Envelope Format**:
+   - `enc:v<version>:<nonce_hex>$<ciphertext_and_tag_hex>`
+   - Allows multi-version key rings to coexist without breaking older stored data.
+3. **Live Key Rotation**:
+   - `KeyRing` supports multiple key versions (`v1`, `v2`, `v3`).
+   - New encryptions always use the `active_version`.
+   - Older ciphertext is transparently decrypted using the corresponding historical key in the key ring.
+   - `reencrypt()` allows zero-downtime lazy or background batch migrations to the latest active key.
+4. **Transparent Decryption & PII Redaction**:
+   - `EncryptedField<T>` wrapper implements `Serialize` and `Deserialize` with Serde.
+   - `Debug` and `Display` implementations automatically redact plaintext (`"[REDACTED_PII]"`) to prevent accidental log leakage.
+   - Explicit `.expose_secret()` or `.decrypt()` method ensures deliberate access.
+5. **Access Audit Trail**:
+   - Every encryption, decryption, failed authentication, key rotation, and re-encryption emits a structured security audit event.
+   - Captures timestamp, operation (`ENCRYPT`, `DECRYPT`, `KEY_ROTATION`, `REENCRYPT`, `DECRYPT_FAILED`), target field name, key version, actor ID, and operation duration in microseconds.
+6. **High Performance (<5% Overhead)**:
+   - Microsecond-level AES-256-GCM execution (sub-10 µs per field).
+   - Minimal allocations and hardware AES-NI acceleration where available.
+
+## Architecture
+
+```mermaid
+flowchart LR
+    Plaintext[PII Plaintext] -->|AES-256-GCM + Nonce| Encrypt[PiiDataProtector.encrypt]
+    Encrypt -->|enc:v1:nonce$ciphertext| Database[(PostgreSQL Storage)]
+    Database -->|enc:v1:nonce$ciphertext| Decrypt[PiiDataProtector.decrypt]
+    KeyRing[(KeyRing v1, v2)] -->|Lookup Key by Version| Decrypt
+    Decrypt --> PlaintextOut[Plaintext PII]
+    Decrypt --> Audit[EncryptionAuditLogger]
+    Encrypt --> Audit
+```
+
+## Key Rotation Workflow
+
+1. Generate a new 256-bit encryption key.
+2. Register and promote the new key in the `KeyRing`:
+   ```rust
+   let new_key = EncryptionKey::generate();
+   key_ring.rotate_key("v2", new_key);
+   ```
+3. All new records will immediately be written under `v2`.
+4. Existing records stored under `v1` continue to be read transparently.
+5. (Optional) Run background re-encryption worker:
+   ```rust
+   let updated_envelope = protector.reencrypt(&old_envelope, "user.email", Some("migration_worker"))?;
+   ```
+
+## Configuration & Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `ENCRYPTION_KEY_V1` | 64-character hexadecimal representation of 256-bit key for version 1 |
+| `ENCRYPTION_KEY_V2` | 64-character hexadecimal representation of 256-bit key for version 2 |
+| `ENCRYPTION_ACTIVE_KEY_VERSION` | Version string of the active key (e.g. `v1`, `v2`) |

backend/DATA_ENCRYPTION.md

@@ -7,6 +7,7 @@ pub mod metrics;
 pub mod middleware;
 pub mod models;
 pub mod realtime;
+pub mod security;
 pub mod service;
 pub mod telemetry;
 pub mod transaction;

backend/src/lib.rs

@@ -14,6 +14,7 @@ mod models;
 mod realtime;
 mod service;
 mod orchestrator;
+mod security;
 mod telemetry;
 mod validators;