Skip to content

backup-restore.md

Latest commit

 

History

History
270 lines (218 loc) · 8.53 KB

backup-restore.md

File metadata and controls

270 lines (218 loc) · 8.53 KB
Raw

Backup + Restore

LeanNodes stores all persistent state in the configured Store backend. DynamoDB uses a single table (leannodes-state by default). Postgres uses one logical table, leannodes_store, in the configured database. This doc covers the three things operators need to know:

  1. What's protected by default — Point-in-Time Recovery (PITR)
  2. How to export — periodic on-demand snapshots to S3
  3. How to restore — both "data corruption" and "table vaporised" scenarios, with DR-drill checklist

1. Default protection — Point-in-Time Recovery (PITR)

DynamoDB

When the orchestrator --bootstrap runs (or the Helm post-install Job fires), it enables PITR on the table. PITR lets you restore the table to any second in the last 35 days as a fresh new table. This is the difference between "lost an hour" and "lost everything" when someone runs DROP with no WHERE.

Verify PITR is on:

aws dynamodb describe-continuous-backups \
  --table-name leannodes-state \
  --region us-east-1

Postgres

For Postgres deployments, use your platform's managed backup feature or a scheduled pg_dump/WAL archive policy. At minimum, production installs should verify:

  • automated backups are enabled for the database/cluster;
  • point-in-time restore is available for the retention window your team requires;
  • the leannodes_store table is included in restore drills;
  • the DSN Secret is backed up or recreated by your secret manager.

Expected:

{
  "ContinuousBackupsDescription": {
    "ContinuousBackupsStatus": "ENABLED",
    "PointInTimeRecoveryDescription": {
      "PointInTimeRecoveryStatus": "ENABLED",
      "EarliestRestorableDateTime": "<35 days ago>",
      "LatestRestorableDateTime": "<~5 seconds ago>"
    }
  }
}

If PointInTimeRecoveryStatus is DISABLED, either:

  • Your IAM policy is missing dynamodb:UpdateContinuousBackups (see deploy/iam/)
  • You're on dynamodb-local (PITR is AWS-only; local dev uses a named docker volume instead — see make ddb-local-backup)

Cost: PITR adds ~$0.20/GB-month on DynamoDB. On a small dev/QA cluster the whole table rarely crosses 10 MB — PITR cost is in the fractional cents.

2. On-demand export to S3 (recommended for quarterly archives)

PITR covers 35 days. For longer retention (compliance, pre- migration checkpoint, before-risky-experiment baseline), export to S3:

# One-off export; completes asynchronously in ~10-20 minutes
aws dynamodb export-table-to-point-in-time \
  --table-arn arn:aws:dynamodb:us-east-1:123456789012:table/leannodes-state \
  --s3-bucket leannodes-backups \
  --s3-prefix "exports/$(date -u +%Y%m%d-%H%M%S)" \
  --export-format DYNAMODB_JSON \
  --region us-east-1

What gets written: newline-delimited JSON under the prefix, plus a manifest. Encrypted with AWS-managed KMS by default; pass --s3-sse-algorithm KMS --s3-sse-kms-key-id <arn> for a customer- managed key.

Schedule it: add a weekly or monthly EventBridge rule pointing to a Lambda that runs the export. Example Terraform:

resource "aws_cloudwatch_event_rule" "leannodes_export" {
  schedule_expression = "cron(0 3 ? * SUN *)"  # Sundays 3am UTC
}
# ... target a Lambda that runs the AWS CLI export command above.

Retention: 90 days is a sensible default for a dev/QA cost optimizer. For SOC2 environments, 1 year is typical. Expire via an S3 lifecycle rule.

3. Restore procedures

Scenario A — "I deleted the wrong flow / accidentally dropped data"

Most common. Use PITR to restore to a specific point in time AS A NEW TABLE, then cut over.

# 1. Restore to a new table at the target timestamp.
aws dynamodb restore-table-to-point-in-time \
  --source-table-name leannodes-state \
  --target-table-name leannodes-state-restored \
  --restore-date-time 2026-04-24T18:30:00Z \
  --region us-east-1

# 2. Wait for the restored table to go ACTIVE (usually 10-30 min
#    depending on table size).
aws dynamodb wait table-exists \
  --table-name leannodes-state-restored \
  --region us-east-1

# 3. Verify — spot-check specific rows or do a count.
aws dynamodb scan \
  --table-name leannodes-state-restored \
  --select COUNT \
  --region us-east-1

Cutover options:

  • Hot cutover (downtime): stop orchestrator, rename tables (leannodes-state → leannodes-state-broken, leannodes-state-restored → leannodes-state), restart orchestrator.
  • Selective restore: query the restored table for the lost rows, write them back into production via the orchestrator API (PUT /flows/{id} for flows, or ACL grant API for ACLs). Use this for "one flow went missing" cases — faster than a full cutover.

Scenario B — "The table is completely gone"

Rare; usually someone with admin IAM ran DeleteTable manually. Same procedure as A, but there's no source table, so you restore from an S3 export (from §2).

aws dynamodb import-table \
  --s3-bucket-source '{
    "S3Bucket": "leannodes-backups",
    "S3KeyPrefix": "exports/20260424-030000"
  }' \
  --input-format DYNAMODB_JSON \
  --table-creation-parameters '{
    "TableName": "leannodes-state",
    "BillingMode": "PAY_PER_REQUEST",
    "AttributeDefinitions": [
      {"AttributeName":"pk","AttributeType":"S"},
      {"AttributeName":"sk","AttributeType":"S"},
      {"AttributeName":"gsi1pk","AttributeType":"S"},
      {"AttributeName":"gsi1sk","AttributeType":"S"},
      {"AttributeName":"gsi2pk","AttributeType":"S"},
      {"AttributeName":"gsi2sk","AttributeType":"S"}
    ],
    "KeySchema": [
      {"AttributeName":"pk","KeyType":"HASH"},
      {"AttributeName":"sk","KeyType":"RANGE"}
    ],
    "GlobalSecondaryIndexes": [ ... ]
  }' \
  --region us-east-1

The exact GSI definitions live in internal/store/ddb/bootstrap.go:createTable.

After the import, run orchestrator --bootstrap to re-enable TTL

  • PITR on the restored table.

Scenario C — "The AWS region is on fire"

Multi-region restore is out of scope for v1 (single-region by design; see PRODUCT.md). If AWS declares a region down:

  1. Pick a healthy region.
  2. Use S3 cross-region replication on the exports bucket (it should be on already — aws s3api get-bucket-replication).
  3. Import from the replicated bucket in the new region using Scenario B's procedure.
  4. Redeploy the Helm chart pointed at the new region (AWS_REGION + LEANNODES_DDB_TABLE).

A proper multi-region story (DDB global tables, active/active orchestrators) is on the post-v1 roadmap.

4. Local dev (dynamodb-local)

PITR doesn't exist on dynamodb-local. For local development we ship persistent Docker volume tooling instead:

# Snapshot the local DDB volume before a risky experiment.
make ddb-local-backup
# Writes /tmp/leannodes-ddb-backup-<ts>.tar.gz

# Restore from the tarball.
make ddb-local-restore FROM=/tmp/leannodes-ddb-backup-20260425-103000.tar.gz

# Nuclear option (destroys everything).
make ddb-local-wipe

See Makefile for the mechanics. The -wipe target requires typing wipe to confirm — belt-and-braces after the 2026-04-25 incident where a casual docker rm -f deleted every flow.

5. DR drill — do this quarterly

Book an hour with someone who hasn't run a restore before. Use a non-production cluster (or a scratch DDB table). They should complete the following without reading this doc:

  • Verify PITR is enabled (describe-continuous-backups)
  • Restore the table to a specific timestamp (restore-table-to-point-in-time)
  • Verify the restored table has the expected rows
  • Do a cutover (rename + restart orchestrator in a scratch deployment)
  • Undo — delete the scratch tables

If they get stuck, that's the section of THIS doc that needs improvement. Fix it.

6. IAM permissions

The orchestrator's IRSA role needs these for bootstrap + backup:

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:DescribeTable",
    "dynamodb:CreateTable",
    "dynamodb:UpdateTimeToLive",
    "dynamodb:DescribeTimeToLive",
    "dynamodb:UpdateContinuousBackups",
    "dynamodb:DescribeContinuousBackups"
  ],
  "Resource": "arn:aws:dynamodb:*:*:table/leannodes-*"
}

Restore operations require additional permissions on a SEPARATE IAM principal (a human operator / CI job), not the orchestrator's runtime role:

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:RestoreTableToPointInTime",
    "dynamodb:ExportTableToPointInTime",
    "dynamodb:ImportTable",
    "dynamodb:ListExports",
    "dynamodb:ListImports",
    "dynamodb:DescribeExport",
    "dynamodb:DescribeImport"
  ],
  "Resource": "arn:aws:dynamodb:*:*:table/leannodes-*"
}