Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ DynamoDB, zero errors. See [Correctness](#correctness).

| Failure mode | Behaviour |
|-------------|-----------|
| **DynamoDB slow or partially down** | The rate-limit store is wrapped in a bulkhead (100 concurrent calls, 500 ms max wait), a process-wide circuit breaker (20 failures to open, 30 s reset, 3 half-open calls), a retry policy for SDK and I/O errors (3 retries, 100 ms base, 2x backoff, 10 s cap), and a 2 s timeout per check. A timed-out check is not retried: the bound covers the whole check. When a check reaches no decision (the breaker is open, the bulkhead is full, or the store times out or fails after retries), `DEGRADATION_MODE` decides: `reject-all` (default, safe for payments) or `allow-all` (for AI infrastructure where availability wins). Every degraded decision increments `gate_degraded_total`. `GET /v1/ratelimit/status` answers `503` when the store cannot be read, rather than reporting a full bucket. |
| **DynamoDB slow or partially down** | The rate-limit store is wrapped in a bulkhead (100 concurrent calls, 500 ms max wait), a process-wide circuit breaker (20 failures to open, 30 s reset, 3 half-open calls), a retry policy for SDK and I/O errors (3 retries, 100 ms base, 2x backoff, 10 s cap), and a 2 s timeout per check. A timed-out check is not retried: the bound covers the whole check. When a check reaches no decision (the breaker is open, the bulkhead is full, or the store times out or fails after retries), `DEGRADATION_MODE` decides: `reject-all` (default, safe for payments) or `allow-all` (for AI infrastructure where availability wins). Every degraded decision carries `X-Gate-Degraded: true` and increments `gate_degraded_total`. `GET /v1/ratelimit/status` answers `503` when the store cannot be read, rather than reporting a full bucket. |
| **A corrupt limiter or quota item** | It fails closed and self-heals. The store replaces the item, conditioned on its raw stored version, with the most conservative valid state: an empty token bucket, a full leaky bucket or sliding window, an exhausted quota window starting now. The key refuses until it refills, drains, or the window ends. Each read counts `CorruptStateRead`, which has an alarm. A corrupt idempotency record answers `503`, since whether its operation ran is unknowable. |
| **Instance crash** | No in-process state. The next instance reads current DynamoDB state and continues correctly. |
| **Kinesis failure** | Events go into a bounded in-memory queue (10,000, drop-oldest) that a background fiber drains to Kinesis. A failed publish is retried once, then dropped. Both an eviction from a full queue and a failed publish are counted, as `gate_events_dropped_total` and CloudWatch `DroppedKinesisEvent{reason=queue_full\|publish_failed}`. The request path never waits on Kinesis, and `/ready` reports a Kinesis fault as `degraded` without taking the task out of service. |
Expand Down Expand Up @@ -187,7 +187,8 @@ curl -s -X POST http://localhost:8080/v1/ratelimit/check \
"retryAfter": null,
"limit": 1000,
"resetAt": "2026-09-14T10:30:05Z",
"message": null
"message": null,
"error": null
}
```

Expand Down
108 changes: 71 additions & 37 deletions docs/API.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Middleware, outermost first:
2. **[CorrelationIdMiddleware](../src/main/scala/observability/CorrelationIdMiddleware.scala).** It takes `X-Request-Id` or generates a UUID, keeps it in an `IOLocal` for audit events, and echoes it on the response.
3. **[TracingMiddleware](../src/main/scala/observability/TracingMiddleware.scala).** One server span per request, with `http.method`, `http.target`, and `http.status_code`.
4. **Public routes** `/health` and `/ready`, then the dashboard routes when enabled. Neither goes through authentication.
5. **[ApiKeyAuth.middleware](../src/main/scala/security/ApiKeyAuth.scala).** It reads the key from `Authorization: Bearer <key>`, `Authorization: ApiKey <key>`, or `X-Api-Key`. A missing or unknown key gets a bare 401. A known key then passes the per-instance auth throttle: `security.authentication.rate-limit-per-minute` (1,000) requests per key, in a minute that starts at the key's first request. Over it, the answer is 429 with `Retry-After`, not 401, because the two need opposite client behavior: fix the credentials, or back off. Unknown keys are counted per source (`failed-attempts-per-minute`, 20); a source over it gets the same 429 with no lookup, valid key or not, so a guess that lands does not get through. Behind the ALB (`trust-forwarded-for`, set by Terraform) the source is the last `X-Forwarded-For` entry; otherwise it is the connecting address.
5. **[ApiKeyAuth.middleware](../src/main/scala/security/ApiKeyAuth.scala).** It reads the key from `Authorization: Bearer <key>`, `Authorization: ApiKey <key>`, or `X-Api-Key`. A missing or unknown key gets a 401 with an `unauthorized` error body. A known key then passes the per-instance auth throttle: `security.authentication.rate-limit-per-minute` (1,000) requests per key, in a minute that starts at the key's first request. Over it, the answer is 429 with `Retry-After`, not 401, because the two need opposite client behavior: fix the credentials, or back off. Unknown keys are counted per source (`failed-attempts-per-minute`, 20); a source over it gets the same 429 with no lookup, valid key or not, so a guess that lands does not get through. Behind the ALB (`trust-forwarded-for`, set by Terraform) the source is the last `X-Forwarded-For` entry; otherwise it is the connecting address.
6. **Permission check** (`ApiKeyAuth.requirePermission`). A key without the route's permission gets 403 before any state is touched.

An unauthenticated request to an unknown path therefore gets 401; an authenticated one gets 404.
Expand Down Expand Up @@ -138,7 +138,7 @@ All three stores run the same loop ([ADR-004](adr/004-occ-over-pessimistic-locki
When the wrapped call still fails, whether from an open breaker, a full bulkhead, an error, or a timeout, the check does not fail. It answers from `resilience.degradation-mode` ([GracefulDegradation](../src/main/scala/resilience/GracefulDegradation.scala)) and counts `RateLimitDegraded{reason}`:

- `reject-all` (the default in `application.conf` and Terraform) answers 429 with `Retry-After: 60`.
- `allow-all` answers 200 with 100 tokens remaining. Downstream spend is unbounded while degraded.
- `allow-all` answers 200 with the profile's capacity remaining. Downstream spend is unbounded while degraded.

`GET /v1/ratelimit/status/{key}` reads one item through the same wrapper, but a failure propagates instead of degrading, and the API answers 503 `storage_unavailable`. It used to report a failed read as a key never seen, so an outage showed every bucket full. A key that really was never seen reads as full capacity.

Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/static/dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -875,7 +875,7 @@ <h2 class="config-title">Configuration</h2>
if (!res.ok) {
const error = await res.json();
showConfigMessage(
error.error || "Failed to update configuration",
error.message || "Failed to update configuration",
"error",
);
return;
Expand Down
12 changes: 6 additions & 6 deletions src/main/scala/Main.scala
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import scala.concurrent.duration.*

import org.http4s.ember.server.EmberServerBuilder
import org.http4s.server.{Router, Server}
import org.http4s.server.Server
import org.typelevel.log4cats.Logger
import org.typelevel.log4cats.slf4j.Slf4jLogger
import org.typelevel.otel4s.oteljava.OtelJava
Expand Down Expand Up @@ -141,11 +141,11 @@ object Main extends IOApp:

appWithCorrelation = CorrelationIdMiddleware
.middleware(correlationLocal)(routes.routes)
// ErrorHandling turns a MessageFailure into a 4xx (400 for a body that
// is not JSON, 422 for one that does not match the schema); without it
// an undecodable body surfaces as an empty 500.
httpApp: org.http4s.HttpApp[IO] = org.http4s.server.middleware
.ErrorHandling.httpApp(Router("/" -> appWithCorrelation).orNotFound)
// toHttpApp turns a MessageFailure into a 4xx (400 for a body that is
// not JSON, 422 for one that does not match the schema); without it an
// undecodable body surfaces as an empty 500.
httpApp: org.http4s.HttpApp[IO] = Routes
.toHttpApp(appWithCorrelation, summon[Logger[IO]])

_ <- Resource.make(Async[IO].unit)(_ =>
summon[Logger[IO]]
Expand Down
44 changes: 27 additions & 17 deletions src/main/scala/api/DashboardApi.scala
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ class DashboardApi[F[_]: Async](

// GET /dashboard — serve the single-page HTML dashboard from classpath
case req @ GET -> Root / "dashboard" => StaticFile
.fromResource[F]("static/dashboard.html", Some(req))
.getOrElseF(NotFound("Dashboard resource not found"))
.fromResource[F]("static/dashboard.html", Some(req)).getOrElseF(NotFound(
ApiError.body(ApiError.NotFound, "Dashboard resource not found"),
))

// GET /dashboard/api/config — bucket configuration for the UI
case GET -> Root / "dashboard" / "api" / "config" =>
Expand Down Expand Up @@ -110,7 +111,9 @@ class DashboardApi[F[_]: Async](
"message" -> Json.fromString("Configuration updated successfully"),
))
yield response).handleErrorWith(error =>
BadRequest(Json.obj("error" -> Json.fromString(error.getMessage))),
// It answered `{"error": reason}` alone, with the reason where every
// other route puts a code.
BadRequest(ApiError.body(ApiError.ValidationError, error.getMessage)),
)

// POST /dashboard/api/check — consume one token from the demo bucket
Expand All @@ -121,14 +124,14 @@ class DashboardApi[F[_]: Async](
now <- Clock[F].realTime.map(d => Instant.ofEpochMilli(d.toMillis))
_ <- publishDashboardDecision(decision, now)
response <- decision match
case RateLimitDecision.Allowed(tokensRemaining, resetAt) =>
case RateLimitDecision.Allowed(tokensRemaining, resetAt, _) =>
Ok(Json.obj(
"allowed" -> Json.True,
"tokensRemaining" -> Json.fromInt(tokensRemaining),
"limit" -> Json.fromInt(profile.capacity),
"resetAt" -> Json.fromString(resetAt.toString),
))
case RateLimitDecision.Rejected(retryAfter, resetAt) =>
case RateLimitDecision.Rejected(retryAfter, resetAt, _) =>
// Always 200 so dashboard JS can read the body without error handling.
Ok(Json.obj(
"allowed" -> Json.False,
Expand Down Expand Up @@ -174,7 +177,10 @@ class DashboardApi[F[_]: Async](
yield ServerSentEvent(data = Some(data.noSpaces))
}.handleErrorWith(error =>
Stream.eval(Async[F].delay(ServerSentEvent(data =
Some(Json.obj("error" -> Json.fromString(error.getMessage)).noSpaces),
Some(
ApiError.body(ApiError.StorageUnavailable, error.getMessage)
.noSpaces,
),
))),
)

Expand All @@ -187,15 +193,18 @@ class DashboardApi[F[_]: Async](
val eventStream = fs2.Stream.repeatEval(queue.take)
.map(ev => ServerSentEvent(data = Some(ev.asJson.noSpaces)))
Ok(eventStream)
case None => NotFound()
case None => NotFound(
ApiError
.body(ApiError.NotFound, "The decision stream is not enabled"),
)
}

private def publishDashboardDecision(
decision: RateLimitDecision,
now: Instant,
): F[Unit] =
val event = decision match
case RateLimitDecision.Allowed(tokensRemaining, _) => RateLimitEvent
case RateLimitDecision.Allowed(tokensRemaining, _, _) => RateLimitEvent
.Allowed(
timestamp = now,
apiKey = "dashboard-demo",
Expand All @@ -205,15 +214,16 @@ class DashboardApi[F[_]: Async](
cost = 1,
tier = "dashboard",
)
case RateLimitDecision.Rejected(retryAfter, _) => RateLimitEvent.Rejected(
timestamp = now,
apiKey = "dashboard-demo",
clientId = "dashboard",
endpoint = "dashboard",
retryAfterSeconds = retryAfter,
reason = "Rate limit exceeded",
tier = "dashboard",
)
case RateLimitDecision.Rejected(retryAfter, _, degraded) => RateLimitEvent
.Rejected(
timestamp = now,
apiKey = "dashboard-demo",
clientId = "dashboard",
endpoint = "dashboard",
retryAfterSeconds = retryAfter,
reason = RateLimitApi.rejectionReason(degraded),
tier = "dashboard",
)
eventPublisher.publish(event).handleError(e =>
logger.warn(s"Dashboard event publish failed: ${e.getMessage}"),
)
Expand Down
49 changes: 25 additions & 24 deletions src/main/scala/api/IdempotencyApi.scala
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,10 @@ class IdempotencyApi[F[_]: Async: Tracer](
checkReq <- request.as[IdempotencyCheckRequest]
response <- checkReq.ttl.filter(_ <= 0) match
// It used to be accepted, and wrote a record that was already expired.
case Some(ttl) => BadRequest(io.circe.Json.obj(
"error" -> io.circe.Json.fromString("validation_error"),
"message" ->
io.circe.Json.fromString(s"ttl must be positive, got $ttl"),
))
case Some(ttl) => BadRequest(
ApiError
.body(ApiError.ValidationError, s"ttl must be positive, got $ttl"),
)
case None => runCheck(checkReq, client, startTime)
yield response).handleErrorWith(storageFailure("check"))

Expand Down Expand Up @@ -128,15 +127,11 @@ class IdempotencyApi[F[_]: Async: Tracer](
.debug(s"Completing idempotency key: $key, client=${client.clientId}")
response <-
if storedBytes > IdempotencyApi.MaxStoredResponseBytes then
PayloadTooLarge {
io.circe.Json.obj(
"error" -> io.circe.Json.fromString("response_too_large"),
"message" -> io.circe.Json.fromString(
s"The response to store is $storedBytes bytes encoded; the limit is ${IdempotencyApi
.MaxStoredResponseBytes}. The key stays pending: store a smaller response, such as a reference to the result, or mark the key failed.",
),
)
}
PayloadTooLarge(ApiError.body(
"response_too_large",
s"The response to store is $storedBytes bytes encoded; the limit is ${IdempotencyApi
.MaxStoredResponseBytes}. The key stays pending: store a smaller response, such as a reference to the result, or mark the key failed.",
))
else
store.storeResponse(
TenantKey(client.clientId, key),
Expand All @@ -159,6 +154,7 @@ class IdempotencyApi[F[_]: Async: Tracer](
status = "conflict",
message =
Some("Could not store response - key may not exist or is not pending"),
error = Some(IdempotencyApi.NotPendingCode),
).asJson,
),
)
Expand Down Expand Up @@ -190,6 +186,7 @@ class IdempotencyApi[F[_]: Async: Tracer](
status = "conflict",
message =
Some("Could not mark failed - key may not exist or is not pending"),
error = Some(IdempotencyApi.NotPendingCode),
).asJson,
)
yield response).handleErrorWith(storageFailure("fail"))
Expand All @@ -201,20 +198,16 @@ class IdempotencyApi[F[_]: Async: Tracer](
private def storageFailure(op: String): Throwable => F[Response[F]] =
case e: CorruptIdempotencyRecordException => logger
.error(e)(s"Corrupt idempotency record for key=${e.key}") *>
ServiceUnavailable(io.circe.Json.obj(
"error" -> io.circe.Json.fromString("storage_corruption"),
"message" ->
io.circe.Json
.fromString(s"Idempotency record corrupted: ${e.detail}"),
ServiceUnavailable(ApiError.body(
"storage_corruption",
s"Idempotency record corrupted: ${e.detail}",
))
case e: MessageFailure => Async[F].raiseError(e)
case e => logger.error(e)(s"Idempotency $op failed") *>
(if op == "check" then countCheck("error") else Async[F].unit) *>
ServiceUnavailable(io.circe.Json.obj(
"error" -> io.circe.Json.fromString("storage_unavailable"),
"message" -> io.circe.Json.fromString(
s"Idempotency store failed during $op; retry the request",
),
ServiceUnavailable(ApiError.body(
ApiError.StorageUnavailable,
s"Idempotency store failed during $op; retry the request",
))

private def countCheck(result: String): F[Unit] = metricsPublisher
Expand Down Expand Up @@ -268,6 +261,7 @@ class IdempotencyApi[F[_]: Async: Tracer](
originalResponse = None,
message =
Some("Request body does not match the original request for this idempotency key"),
error = Some(IdempotencyApi.ConflictCode),
).asJson,
)

Expand Down Expand Up @@ -363,6 +357,9 @@ case class IdempotencyCheckResponse(
originalResponse: Option[OriginalResponse] = None,
firstSeenAt: Option[String] = None,
message: Option[String] = None,
// Set with `message` on the 409, like every other error body. `status`
// still carries the outcome.
error: Option[String] = None,
)

case class OriginalResponse(
Expand All @@ -381,9 +378,13 @@ case class IdempotencyUpdateResponse(
idempotencyKey: String,
status: String,
message: Option[String] = None,
error: Option[String] = None,
)

object IdempotencyApi:
val ConflictCode = "idempotency_conflict"
val NotPendingCode = "not_pending"

/** The largest stored response, in bytes of its encoded JSON. A DynamoDB item
* holds at most 400 KB, and the response is stored inline in the record
* (replay is not streamed), so a larger one failed the write and answered
Expand Down
Loading
Loading