Building on AWS · guide 10 of 13

Disaster Recovery and Backups for Startups on AWS: Inventory, Tiers, Restore Checks and Break-Glass

8981 words · about 45 minutes · all guides

Abstract. Most startups can show you a green “backup succeeded” status. Far fewer have restored one of those backups in the last month, and fewer still could say which of their assets cannot be restored at all. This article is for senior DevOps, SRE and platform engineers at AWS-native startups, and especially at crypto infrastructure companies, where some of the most important assets (signing keys, on-chain ownership, user-signed authorisations) have no backup by design. It covers six things. The first is an asset inventory with criticality tiers and a recoverability class. The second is a recovery dependency DAG. The third is RTO/RPO targets that say whether they were measured or guessed. The fourth is the choice between AWS Backup and native snapshots. The fifth is automated restore verification that actually restores and checks real data. The sixth is break-glass playbooks built around irreversible decision points. Along the way it covers freshness alerting, broker topology export, Kubernetes PersistentVolume traps and drill logs. It also argues for writing “never drilled” when that is true. The lessons come from a real production platform, generalised and anonymised. Most of them are cases where a control looked correct and was quietly broken.

1. The problem space

1.1 Why DR at a startup goes wrong

DR work at a startup usually fails in one of three ways:

Backups exist, but nobody has restored them. RDS automated backups are on by default, ElastiCache snapshots are one checkbox, and S3 versioning is one line of Terraform. The backup half of DR is nearly free on AWS. The restore half is not. A restore needs IAM permissions that nobody has exercised, network placement, credentials, cutover steps and application reconnect behaviour. All of that rots while nobody is looking. The Google SRE book puts it this way: “No one really wants to make backups; what people really want are restores” [18].
The asset list is whatever someone remembered. A team usually knows about its primary database. It forgets the cache that slowly became a system of record, the hand-created EBS volume holding a 30-hour index, the KMS key created in the console eighteen months ago, and the broker that has no backup feature at all.
The documentation is more confident than reality. A “monthly DR drill” and a “cross-region replica” that were only ever intentions now read as current fact. A false claim is worse than a blank, because nobody checks a claim that looks authoritative.

1.2 Forces specific to startups

Team size. There are two to six infrastructure engineers, and usually no dedicated DR owner. Anything that needs a human to remember it weekly will stop happening.
Cost sensitivity. Cross-region replicas, warm standby and a second AWS account all cost money and add operational load. AWS’s own guidance says that for a workload that is highly available within a Region, plain backup-and-restore may be all the DR you need [3]. Start there, and spend more only when you have a reason.
Partner and auditor pressure. SOC 2, enterprise customer questionnaires and exchange or custody partners will ask for your RTO/RPO, the date of your last restore test, and whether your backups are immutable. What they want is evidence, not a policy document. The AWS Well-Architected Reliability Pillar says the same in two best practices: back up data automatically, and “perform periodic recovery of the data to verify backup integrity and processes” (REL09-BP03/04) [1]. It also says to test the DR implementation itself (REL13-BP03) [2].
A single account. Many startups run production, its backups, its KMS keys and its Terraform state in one AWS account and one Region. That makes account compromise the scenario that matters most, and it is also the one with the least coverage.

1.3 Forces specific to crypto

Crypto infrastructure adds a whole asset class that cannot be restored, only succeeded:

Non-exportable KMS signing keys. An ECC_SECG_P256K1 or ECC_NIST_EDWARDS25519 key with SIGN_VERIFY usage behind a hot wallet, relayer, deployer or fee collector has no backup. Deleting a KMS key is irreversible once the 7–30 day waiting period ends [11]. If that key is the on-chain owner of a proxy, a treasury or a counterfactual account factory, deleting it can freeze funds permanently.
Derived addresses. Funds sit at an address, and the address is a function of the key. If you cannot get a key’s public key, you cannot even list the addresses you need to rescue.
User-signed authorisations. Session-key enablement signatures, permits and off-chain orders were produced by a user’s wallet. You cannot regenerate them. If they live only in a database or cache, that store is Tier 0, whatever its name says.
Residual funds in decommissioned wallets. A staging environment or retired service often still holds balances on several chains. If you tear down its keys before sweeping, the funds are lost. If you leave the keys in place, you have unmanaged signing authority sitting in the account.
Industry standards assume a seed backup exists. The CryptoCurrency Security Standard (CCSS) says at Level I that “a backup of the cryptographic key/seed must exist” and requires a documented Key Compromise Protocol, with periodic testing at Level III [21]. A non-exportable HSM-backed key cannot meet the first requirement as written. You need a different design, covered in §4.5.

2. Reference design overview

The design splits the work into separate artifacts, each with one job, so they do not drift into each other:

Artifact	Job	Lifecycle
Runbook	Diagnose and mitigate a system that still exists. Indexed by symptom. It stays the front door for every alert.	Edited constantly
DR router	One page: a scenario index linking to the playbook for each scenario, plus a “last drilled” column	Edited when a playbook or drill lands
Asset inventory	Every store, key, on-chain entity and platform dependency, with tier, recoverability, backup, restore path, last verified date, edges and owner. Contains the recovery DAG.	Timeless facts only. Replace a claim when it changes. Never append.
RTO/RPO table	The target contract per store, with each number marked measured or provisional	Updated only from drill logs
Break-glass playbooks	One per scenario. Covers incident sequencing, irreversibility gates and a verification checklist. Wraps the runbook’s restore mechanics and does not repeat them.	Edited after drills
Drill logs	Dated and append-only. Commands, trimmed output, wall-clock time per phase, deviations.	Never edited, except to fix an error
Automated restore checks	Weekly restore-and-validate of every Tier 0/1 store, with failure and freshness alerts	Code

Keep these separate on purpose. When the inventory holds drill results, it fills up with dated prose that is stale within a month. When a playbook repeats the restore commands, the two copies drift apart. The RTO table belongs in the runbook, because backup-integrity alerts can then link straight to a row via runbook_url.

How the pieces fit together: symptoms route to runbooks, runbooks route to disaster-recovery playbooks, and automated restore checks feed back into both. full size ↗

The line between the runbook and break-glass needs to be stated explicitly. Break-glass starts when the store or platform is gone or untrusted and the next action is irreversible. Examples are restoring over a live endpoint, accepting data loss since time T, or retiring a key. Most SEV1 incidents never cross that line.

3. Reference design in detail

3.1 The asset inventory

Every entry uses the same fields in the same order, so the file can be grepped and reviewed:

# One entry per asset. A blank field is a finding, not an option.
- asset: rds/payments-db
  holds: "orders, users, sessions: money-moving core state"
  tier: 0                     # 0-3, see below
  recoverability: B           # R rebuildable | B restorable from backup | I irreplaceable
  backup: "automated snapshots + PITR, retention 30d"
  rpo_target: "<=5m"
  restore_path: "runbook#rds-restore, dr/BG-1"
  last_verified: "2026-xx-xx · weekly restore check exec arn:... · 17.1M rows sampled, OK"
  depends_on: [network, kms/rds-cmk]
  feeds: [payments-api, ledger-svc, analytics-fdw]
  loss_impact: "payments-api FAILS, ledger-svc FAILS (see service map)"
  owner: unassigned           # say so honestly; see below

Tiers are defined by what loss costs the business, not by technology:

Tier	Definition	Typical members
0	Loss is permanent (funds, on-chain identity, user identity), or it halts money movement	Core transactional DB, signing keys, treasury/owner keys, user-signed authorisations, in-flight funds-adjacent messages
1	Customer-facing degradation, recoverable with time and effort	Secondary DBs, long-rebuild indexers, shared caches carrying operational state
2	Internal tooling, forensics, compliance	Metrics history, audit-log exports, BI metadata, Grafana DB
3	Disposable or regenerable by definition	Pure caches with TTLs, build artifacts, WAL buffers whose durable copy is in S3

Tier 3 is listed explicitly so that “not in the inventory” never has to be interpreted. If an asset is missing, nobody has looked at it yet. That should never mean the team decided it didn’t matter.

Recoverability is a separate axis from tier, and it is the one most inventories leave out:

R (rebuildable): recreated from source, IaC or upstream. An example is an analytics mart rebuilt from the primary DB plus the next dbt run.
B (restorable): a backup exists and has a restore path.
I (irreplaceable): no restore is possible. For these assets the entry must name prevention controls and a succession plan in place of a restore path. Signing keys, raw private keys in a password manager, and messages in flight on a broker with no backup feature all fall here.

A Tier 0 / I asset is the most dangerous combination, and the inventory should make those entries easy to spot.

Fields that make an inventory honest:

last_verified is date · mechanism · result, and “never” is a valid value. Generate it from data where you can, for example by linking the Step Functions execution ARN. A hand-written “confirmed weekly” is exactly the kind of claim that stays false for months (§5.1).
owner: unassigned is better than a made-up owner. If your org has one engineering team and no CODEOWNERS, write that down. A plausible but fictional owner looks authoritative, so nobody chases it, and at 3am someone pages a team that doesn’t exist.
Verify live state, not just the config. One inventory said “no stateful workload uses a PersistentVolume.” A live describe-volumes found five, and three of them were unencrypted despite a StorageClass that said encrypted: "true" (§5.4).

Getting the first inventory done. Combine three sweeps. First, grep the IaC for every stateful resource type (aws_db_instance, aws_elasticache_*, aws_mq_broker, aws_kms_key, aws_s3_bucket, kubernetes_persistent_volume*, aws_secretsmanager_secret). Second, run an account-wide live sweep in every Region that has customer resources (kms list-keys / list-aliases, ec2 describe-volumes, rds describe-db-instances, backup list-backup-plans). Third, diff the two. The diff is where the findings are. One team found five enabled secp256k1 keys, with names suggesting treasury and deployer roles, that no line of Terraform referenced, and a single ed25519 signing key shared by dev and prod.

3.2 The recovery DAG

The inventory’s depends_on and feeds edges together form a graph. Write down the order for a full-stack rebuild, and list the cycles explicitly so nobody gets stuck going round one mid-incident:

The recovery order for a full-stack rebuild, from network and state bucket up to the cluster. full size ↗

Cycles we have seen in practice:

Terraform-state recovery needs the git host. It needs the Terraform config itself and the import commands. If the git org is lost at the same time as the state bucket, there is no clean starting point. Mitigation: keep a recent mirror of the infra repo somewhere outside the git host.
GitOps sync needs the git host. You can rebuild a cluster during a git outage, but ArgoCD/Flux will schedule nothing useful until git is back.
The monitoring tool’s Terraform credential lives in the monitoring tool’s database. A snapshot restore brings it back. A from-scratch rebuild does not, and someone has to mint a new token by hand before terraform apply can authenticate.
Remote-state reverse edges. If environment A reads B’s state and B reads A’s, a from-scratch rebuild becomes a hand-ordered series of -target applies. If that order isn’t written down, it effectively doesn’t exist.

3.3 RTO/RPO per tier, with provenance

AWS frames RTO and RPO as business decisions, set per workload, and the strategy follows from them (REL13-BP01/02) [2]. The part teams usually skip is saying where each number came from:

Store	Tier	RPO target	RTO	Basis
Core RDS (db.r-class, ~100 GB)	0	≤5 min	~8 min, measured (restore start → `available`, 3 runs)	30-day PITR; RDS ships transaction logs to S3 every five minutes [7]
Deposit RDS (db.t4g.large, 20 GB)	0	≤5 min	6–8 min, measured	Same as above. Small instances restore no faster in practice.
Shared cache (Valkey)	1	Per key class (§3.4.2)	~9 min, measured, 1.3M keys scanned	Daily snapshot, 7-day retention [8]
Broker (managed RabbitMQ)	0 for in-flight	0 messages recoverable: accepted structural risk	Provisional	No native message backup. Topology rebuilt from IaC plus a definitions export.
Terraform state bucket	0	0 for object-level loss	Minutes (object), unbounded (whole bucket)	Versioning [17]; no CRR, accepted
Indexer PV (600 GiB EBS)	1	≤24 h	Provisional, never drilled	DLM daily, crash-consistent
Analytics DB	2	≤24 h	Best effort	Rebuildable from sources

Two rules:

Mark a number “provisional” until a drill log has measured it. Never quote a provisional number to a partner as an SLA.
Measure “restore start → validated”, not “restore start → available”. A freshly restored RDS instance or EBS volume loads blocks lazily from S3. AWS’s documentation says performance “might not be at its fullest until initialization completes” [7]. In one validation run, the first COUNT(*) on a cold restore took more than a minute on a 163k-row sample. That broke a 60-second query timeout twice before anyone recognised the pattern.

3.4 Backup mechanisms per store

3.4.1 Relational databases (RDS)

Native automated backups with PITR are the baseline. Set the following in your database module so that no caller can forget them:

resource "aws_db_instance" "this" {
  # ...
  backup_retention_period   = var.tier == 0 ? 30 : 7   # 35 is the RDS max
  deletion_protection       = true
  skip_final_snapshot       = false
  final_snapshot_identifier = "${var.identifier}-final"
  copy_tags_to_snapshot     = true
  multi_az                  = var.tier <= 1

  lifecycle {
    # Cutover restores pass snapshot_identifier via -var; see §5.5 for why
    # this ignore_changes makes "-var alone" a silent no-op.
    ignore_changes = [snapshot_identifier]
  }
}

Multi-AZ is an availability feature, not a backup. It replicates corruption and DROP TABLE just as faithfully as it replicates good writes. AWS’s DR guidance makes the same point about continuous replication generally: it “may not protect against disaster events such as data corruption or malicious attack … as well as point-in-time backups” [3].

3.4.2 Caches (ElastiCache/Valkey/Redis)

A cache entry in the inventory is not finished until each key prefix is classified. One team’s “cache” turned out to contain:

pure TTL caches (token metadata, quote caches), which need no target;
third-party webhook subscription handles with no TTL, where re-enrolling means a vendor API call;
a funds-adjacent pending-resweep set with no TTL, no other source of truth and no alert if it gets stranded;
admin-toggled kill switches with no TTL and no other source of truth.

The classification changes the restore procedure. A restored snapshot can un-pause a kill switch that someone deliberately set after the snapshot was taken. So the playbook needs an “inspect restored operational state before reconnecting apps” step (HGETALL <service>:kill-switches and compare against intended state).

The other lesson from the same team cost far more. An earlier version of that cache was single-node, with no replica, no AOF and no snapshots. It held user-signed session-enable signatures and client configuration, and nothing else had a copy. The node ran out of memory and was replaced, and every key was wiped. Signatures for accounts not yet deployed on-chain were gone for good, some deposits were stuck until users re-signed, and some inbound deposits could no longer be matched to an account. The fix had two parts. Non-derivable state moved to Postgres, under the 30-day PITR. Then every remaining key class was written into a classification table in the runbook. If a store holds data nobody can regenerate, it is a database, whatever you call it.

3.4.3 Message brokers

Amazon MQ for RabbitMQ has no snapshot or PITR feature. Durability means quorum replication, nothing more. Be clear about what can and cannot be recovered:

Topology (vhosts, users, exchanges, queues, bindings, policies) is recoverable. Terraform should own the vhost, users and exchanges. Each consumer should declare its own queues and bindings on connect, so that a rebuilt broker is repopulated by reconnecting clients. Add a daily definitions export to versioned S3 as a pre-warm and drift safety net. RabbitMQ definitions cover “users, vhosts, queues, exchanges, bindings, runtime parameters” and not message data [15].
In-flight messages are not recoverable. Record this as accepted structural risk and do not leave it as an open gap. Then, per exchange, record what losing in-flight messages means. “Publisher sees a failure and retries” is a very different outcome from “consumer state silently diverges from the chain”.

# Daily export job (CronJob / Lambda). The export contains password hashes:
# RabbitMQ's own docs call user records sensitive [15]. Use SSE-KMS and a
# dedicated CMK, and treat that CMK as load-bearing for broker DR.
curl -fsS -u "$MGMT_USER:$MGMT_PASS" "https://$BROKER/api/definitions" \
  | aws s3 cp - "s3://$BUCKET/definitions/$(date -u +%F).json" \
      --sse aws:kms --sse-kms-key-id "$CMK_ARN"

One side effect is easy to miss: the CMK that encrypts the export bucket is now a dependency of broker DR. If someone deletes it, your only broker DR artifact can no longer be read.

3.4.4 S3 and Terraform state

Versioning on every bucket whose data you would want back. HashiCorp’s S3 backend documentation says it is “highly recommended” for state, “to allow for state recovery in the case of accidental deletions and human error” [17]. Single-object recovery is then a non-destructive copy-object of the good version.
Leave buckets unversioned deliberately only when the next build regenerates them, and record that in the inventory.
Object Lock when you need WORM. Governance mode can be bypassed with s3:BypassGovernanceRetention. Compliance mode cannot be bypassed by any user, “including the root user”; the only way out early is deleting the account [10]. Object Lock protects versions. A plain DELETE still adds a delete marker, so you have to look underneath it.
Full loss of the state bucket means re-terraform importing every resource in every environment. That is unbounded effort, but AWS resources keep running meanwhile. Most startups reasonably accept this, provided object-level versioning is in place.

3.4.5 Kubernetes PersistentVolumes

PVs are where DR assumptions break most often (§5.4). For each PV, decide whether it actually needs a backup and, if so, which kind:

PV contents	Right mechanism	Why
WAL/index buffer for a system whose durable data is already in S3 (Loki, Tempo ingesters)	None. Mark it Tier 3 / R.	A rebuild loses minutes of unflushed data
Time-series DB with a native backup tool (VictoriaMetrics, Prometheus TSDB)	App-native backup to S3 (`vmbackup`)	Takes instant snapshots via `/snapshot/create` without stopping the server, and backs up incrementally [16]. It is application-consistent, and S3 storage costs roughly half as much as EBS snapshot storage.
Index/cursor state with a long rebuild time (indexers, substreams sinks)	DLM EBS snapshots [9]	Nothing better exists. Accept crash-consistency and verify after restore that the workload resumed from its cursor and did not quietly start a 30-hour backfill.

resource "aws_dlm_lifecycle_policy" "indexer_pv" {
  description        = "Daily snapshot of the indexer PV"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"
  policy_details {
    resource_types = ["VOLUME"]
    target_tags    = { Name = "indexer-data-prod" }   # this tag IS the backup
    schedule {
      name = "daily"
      create_rule {
        interval      = 24
        interval_unit = "HOURS"
        times         = ["02:30"]
      }
      retain_rule { count = 7 }
      copy_tags = true
      tags_to_add = {
        SnapshotCreator = "dlm"    # filter restores on this stable tag...
        DrPolicy        = "indexer" # ...not on this one, which can be renamed
      }
    }
  }
}

Note the target_tags. If someone hand-creates a replacement volume during an incident and gives it a different Name, that volume is unbacked from that moment on, and no alert fires.

3.5 Automated restore verification

This is the most valuable part of the design, and the part most often skipped.

3.5.1 What “verified” has to mean

A restore check must:

Actually restore the latest automated snapshot, or a PITR point, into a scratch resource. Checking that the snapshot exists, or that the backup job reported success, is not enough.
Connect with real credentials, over the same network path and TLS that a real restore would use.
Assert on data. Sample row counts on named critical tables and on the largest tables by size. Do not sample whatever table sorts first alphabetically (§5.2).
Always tear down, even if a sibling check failed, with a separate reaper as backstop.
Emit a success timestamp that a freshness alert can read (§3.6).
Run unattended on a schedule.

AWS describes the same pattern: periodic restores driven by the SDK, because “data restore from backup is a control plane operation” and a recently restored copy is a hedge against that control plane being unavailable when you need it [3].

3.5.2 Why Step Functions, not a single Lambda

An RDS or ElastiCache restore takes 6–15 minutes, and Lambda times out at 15 minutes. Use Step Functions to orchestrate start_restore → wait/poll → validate → teardown → report, with a short-lived Lambda for each step:

The restore state machine that orchestrates a long-running data store restore. full size ↗

The important detail is Map fail-fast. In an Inline Map state, “the failure of any iteration causes the Map state to fail. All iterations stop when the Map state fails” [13]. In one implementation, a reporting error in one store’s iteration aborted every other in-flight iteration while they were validating. Scratch instances were orphaned twice, and one run checked only 2 of 7 targets. Setting ToleratedFailureCount = 0 does not help, because zero is already the implicit default. What worked:

No iteration ever fails as a state. Each step catches its own errors and records status: failed in the iteration’s output. The report step never throws.
One aggregate state after the Map fails the execution if any store failed, once every store has finished teardown.

{
  "StartAt": "ForEachStore",
  "States": {
    "ForEachStore": {
      "Type": "Map",
      "ItemsPath": "$.targets",
      "MaxConcurrency": 2,
      "ItemProcessor": {
        "StartAt": "StartRestore",
        "States": {
          "StartRestore": {
            "Type": "Task", "Resource": "${fn_arn}",
            "Parameters": { "action": "start_restore", "state.$": "$" },
            "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Teardown" }],
            "Next": "Wait"
          },
          "Wait":     { "Type": "Wait", "Seconds": 60, "Next": "Poll" },
          "Poll":     { "Type": "Task", "Resource": "${fn_arn}",
                        "Parameters": { "action": "poll", "state.$": "$" },
                        "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Teardown" }],
                        "Next": "Ready?" },
          "Ready?":   { "Type": "Choice", "Choices": [
                          { "Variable": "$.status", "StringEquals": "available", "Next": "Validate" },
                          { "Variable": "$.status", "StringEquals": "timeout",   "Next": "Teardown" }],
                        "Default": "Wait" },
          "Validate": { "Type": "Task", "Resource": "${fn_arn}",
                        "Parameters": { "action": "validate", "state.$": "$" },
                        "Catch": [{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Teardown" }],
                        "Next": "Teardown" },
          "Teardown": { "Type": "Task", "Resource": "${fn_arn}",
                        "Parameters": { "action": "teardown", "state.$": "$" },
                        "Next": "Report" },
          "Report":   { "Type": "Task", "Resource": "${fn_arn}",
                        "Parameters": { "action": "report", "state.$": "$" }, "End": true }
        }
      },
      "Next": "CheckAllResults"
    },
    "CheckAllResults": { "Type": "Task", "Resource": "${fn_arn}",
                         "Parameters": { "action": "check_all", "results.$": "$" }, "End": true }
  }
}

3.5.3 Least privilege for the checker and the reaper

The checker can create and delete RDS instances in production, so scope it tightly:

locals {
  # Automated snapshot ARNs look like snapshot:rds:<instance-id>-<timestamp>.
  # Scoping to exactly these sources means a compromised checker cannot restore
  # an arbitrary snapshot in the account.
  restore_src_arns = [for t in local.targets :
    "arn:aws:rds:${var.region}:${local.account}:snapshot:rds:${t.instance_id}-*"]
}

data "aws_iam_policy_document" "checker" {
  statement {
    sid       = "RestoreFromOwnSnapshotsOnly"
    actions   = ["rds:RestoreDBInstanceFromDBSnapshot"]
    resources = concat(local.restore_src_arns,
      ["arn:aws:rds:${var.region}:${local.account}:db:dr-check-*"])
  }
  statement {
    sid       = "MutateScratchOnly"
    actions   = ["rds:DeleteDBInstance", "rds:AddTagsToResource"]
    resources = ["arn:aws:rds:${var.region}:${local.account}:db:dr-check-*"]
  }
}

Two details. First, check exactly which identifier your IaC outputs return. In §5.1, a provider upgrade changed an output from the instance identifier to the immutable resource ID (db-XXXX). The ARN pattern then matched nothing, and every restore was denied. Second, the scratch reaper is a separate scheduled Lambda that deletes anything tagged as a DR scratch resource and older than N hours. Its only mutating permission is on dr-check-*. Describe* has to be account-wide, because RDS describe actions don’t support resource-level permissions, but deletes stay scoped.

A few more practical points:

Size the scratch resource for what you’re testing. For RDS, a db.t4g.medium is enough to prove restorability and data integrity, because you are not testing performance. For a cache, use the same node type as production, read from the module output. A smaller node fails on memory, not on restorability.
Give the scratch resources real auth (AUTH token, TLS, encryption at rest) and a dedicated security group. They hold a copy of production data.
Assert the right thing for each store. A cache whose entire keyspace has 10-minute TTLs will correctly restore to zero keys hours later, because Redis drops already-expired keys when it loads an RDB. One team failed this check four times in a row and spent days ruling out IAM, engine version and parameter groups before finding the cause. For volatile stores, assert that rdb_last_load_keys_expired > 0 (from INFO persistence): that proves the snapshot had data and it then expired.

3.5.4 AWS Backup restore testing as an alternative

AWS Backup now offers managed restore testing. You get scheduled restores of the latest or a random recovery point, PITR points included, for RDS, Aurora, EBS, EC2, EFS, FSx, DynamoDB, S3 and others. There is an optional validation window of 1–168 hours, automatic cleanup of anything tagged awsbackup-restore-test, and an Audit Manager control for “restore time meets target” [5]. Choosing a random recovery point instead of the latest is a good idea for detecting rot in older backups. The gaps for a typical startup are these. ElastiCache is not on the supported list. The data validation is still yours to write, triggered by the restore job’s events. And the feature only covers resources whose backups AWS Backup itself manages. See §4.2.

3.6 Backup freshness alerting

An alert on failure is not enough. The most common setup is “alert on Step Functions ExecutionsFailed > 0, no_data_state = OK”. It cannot detect a check that stopped running, for example because the schedule was disabled, the state machine was deleted, or the EventBridge role broke. In that case you get no data, which by definition means “OK”. One independent review found exactly this. The platform’s only DR alert was that rule, and nothing watched the VictoriaMetrics restore-check gauge, vmbackup failures (which only went to stderr), the broker definitions export, DLM snapshots, or restore-check freshness.

What to add is freshness: “no successful X in N days”, per store. Have each check write a success timestamp:

# At the end of a successful validate(): one gauge per store
cloudwatch.put_metric_data(
    Namespace="DR/RestoreCheck",
    MetricData=[{
        "MetricName": "LastSuccessEpoch",
        "Dimensions": [{"Name": "Store", "Value": store}],
        "Value": time.time(),
    }],
)

Then alert when the timestamp gets too old. In PromQL terms, after scraping or pushing the gauge:

# Weekly job: fire if the last success is more than 8 days old, or if the
# series is missing entirely (absent() covers the "never ran" case).
(time() - max by (store) (dr_restore_check_last_success_epoch)) > 8 * 86400
or
absent(dr_restore_check_last_success_epoch{store="payments-db"})

Rules worth copying:

Freshness rules use no_data = Alerting, and failure-event rules use no_data = OK. A missing success timestamp means something is wrong. A missing failure count usually just means nothing has failed.
For a job that runs weekly, evaluate hourly with a 2-hour lookback and for: 0s. CloudWatch can publish a datapoint after a delay. If the lookback equals the evaluation interval exactly, a late datapoint can fall between two windows and never be seen. With for > 0, a one-off failure event can age out of the window before a second breaching evaluation happens, and the rule then never fires.
Prefer native AWS metrics for the “is it running at all” backstop (AWS/States ExecutionsFailed, AWS/Lambda Errors): they exist as soon as the resource does. Ship any custom metric, verify it flows, then merge the rule that reads it.
Cover every backup producer, not just the restore checks: DLM (EventBridge DLM Policy State Change and snapshot creation failures), AWS Backup job failures, the broker definitions export, audit-log exports and vmbackup. A sidecar that only echos to stderr does not count as monitored.
Monitor the monitor. If the alerting system itself stops evaluating, every DR alert goes quiet with it. Use an external heartbeat, such as a dead-man’s switch pinged every minute.

3.7 Keys and on-chain entities

3.7.1 The on-chain inventory table

For every signing key, record its key ID and Region, the derived address, the provenance of that address, what it controls on-chain, and what happens if the key dies:

Alias	Key ID · Region	Address	Provenance	Controls	If the key dies
`deployer-owner`	`…` · region-a	`0x…`	derived (`GetPublicKey` → keccak) and corroborated in deploy config	`initialOwner` of every CREATE2 proxy	Proxies frozen forever; canonical addresses unreproducible on new chains
`hot-wallet-1`	`…` · region-a	`0x…`	asserted: a dashboard label, not derived	Hot wallet	Relayer stops signing; funded balance stranded
`guardian-signer`	`…` · region-b	unknown	—	Signer behind a guardian contract	Repoint the contract at a new key. Rotatable by design.
`fee-collector`	`…` · region-a	`…` (Solana)	derived	Treasury, signing disabled by policy	Fees stranded

Leave balances out of this table. They change constantly, and the inventory holds timeless facts only. Link to the live dashboard instead.

Provenance matters during an incident. “Derived from the key” and “read from a config file whose label matches the role name” are different levels of certainty, and you don’t want to sweep the wrong wallet.

3.7.2 Prevention controls on irreplaceable keys

resource "aws_kms_key" "signing" {
  description              = "signer: ${var.name}"
  customer_master_key_spec = var.key_spec          # ECC_SECG_P256K1 | ECC_NIST_EDWARDS25519
  key_usage                = "SIGN_VERIFY"
  deletion_window_in_days  = 30                    # the max; the default is also 30 [11]
  multi_region             = var.multi_region      # can ONLY be decided at creation (§4.5)
  policy                   = data.aws_iam_policy_document.signing.json
  tags = {
    DerivedAddress = var.derived_address            # see below
    Tier           = "0"
  }
  lifecycle { prevent_destroy = true }
}

Put these controls in layers:

prevent_destroy plus the maximum 30-day deletion window. This catches the Terraform refactor that renames a module address and plans a destroy-then-create.
An explicit IAM Deny on kms:ScheduleKeyDeletion, kms:DisableKey and kms:PutKeyPolicy on CI deploy roles, even ones with AdministratorAccess, with carve-outs only for environments that legitimately own keys. An explicit Deny always overrides an Allow.
A key policy that denies kms:Sign to anyone except the signer, by principal (an IRSA role) and by source (NAT EIPs). This is a security control, but it has a DR side effect, covered in §5.6.
An EventBridge/CloudTrail alert on ScheduleKeyDeletion and DisableKey, paging at SEV1. The waiting period only helps if a human notices inside it. A key pending deletion “cannot be used in any cryptographic operations” [11], so the alert also catches the outage the change causes.
Write the derived address into the key’s tags. A tag can be read with describe-key from anywhere, even when the network that allows GetPublicKey is gone.

3.7.3 Residual funds and decommissioning

Decommissioning a service that signed transactions is a break-glass procedure, and it needs its own ordered gates:

Capture forensic records: key IDs, aliases, derived addresses and balances on every chain the service touched. Capture them before anything is destroyed. Once the keys and state are gone, the state file may be the last place those IDs were recorded.
Sweep the funds, then verify the balances are zero.
Disable the key. Do not schedule deletion yet. Wait out an observation period while watching CloudTrail for Sign attempts.
Only then schedule deletion.

Common mistakes: deleting the compute but leaving a Secrets Manager entry holding a funded raw wallet key; leaving enabled, unmanaged legacy keys “just in case”; and a staging teardown stuck halfway with live balances on several mainnets.

3.8 Break-glass playbooks

Use one template for every scenario. An engineer who has never seen a given playbook should be able to find the right one and its first action in under 60 seconds.

# BG-n: <scenario>
1. You arrive here from      — runbook sections that escalate here; the fact that justifies crossing the line
2. Incident state            — [ ] already in a declared SEV (if not, declare first)
3. Last drilled / RTO / RPO  — per store; "never" and "provisional" are allowed and expected
4. Preconditions             — access paths, people (named roles), inputs (restore point, key id)
5. Decision points / irreversibility gate   ← the most important section
6. Sequence                  — numbered; owner + estimate + ✅ success criterion per step;
                               link the runbook for mechanics, do not restate them
7. Recovery verification     — store integrity, one box per (f) consumer, alerting back to OK
                               (not NoData), one end-to-end flow, data-loss ledger
8. Aftermath                 — secrets to rotate, silences to expire, checks to re-arm

The irreversibility gate in §5 is the part that pays for itself. Every gate is a checkbox that has to be ticked before the sequence starts. Examples from real playbooks:

RDS: always restore-and-validate into a scratch instance first, even mid-incident; pick the restore point (latest, or a PITR time before the corruption); for a store that deduplicates money movement, the funds owner signs off before cutover, since a restore can re-process deposits already actioned downstream.
Cache: inspect kill switches and funds-adjacent sets before reconnecting.
Broker: sign-off that 0 in-flight messages will be recovered, communicated before starting.
Terraform state: object-level problem (non-destructive, do it now) or genuine bucket loss (full re-bootstrap, never on a hunch)?
Signing key succession: do not schedule deletion of the old key, or revoke its ability to sign, until every on-chain reference and consumer config is confirmed migrated. The sweep runs while the old key still works, and the contract owner confirms the migration mechanism first; never improvise an owner-transfer call against a funds-holding contract.
Password-manager credential: you are replacing, not recovering. Some secret-sync servers issue their credentials file exactly once, and deleting the server revokes every token it issued, so never delete the old one before the new one is proven.

Each playbook should also say what is not an outage. A broken secret-sync operator does not delete the Secrets it already created; pods keep running and only new or changed secrets stop propagating. Urgent, not a page, and knowing that prevents a destructive panic “fix”.

3.9 Drills and drill logs

Each drill produces a dated, append-only log with a fixed shape: header (date, operator, playbook, target), timeline (commands, trimmed output, wall-clock time per phase), verification results against the playbook’s checklist, deviations and findings (each one a ticket plus a playbook edit), and a propagation checklist:

Playbook “last drilled” updated
RTO cell updated, and “provisional” removed if the drill measured a real number
Inventory “last verified” updated
Router index “last drilled” updated

If the results aren’t propagated into those four places, the next reader has no way of knowing the drill happened.

Evidence rules: link execution ARNs over pasted output, make redactions visible, record counts and table names but never row contents.

For scenarios you cannot safely drill live, such as key succession or account compromise, run tabletop exercises using Google’s “Wheel of Misfortune” format: a game master presents a scenario, and the on-call pair talks through what they would do [19]. NIST SP 800-34 calls this mix of tabletop and functional exercises part of a contingency plan’s test, training and exercise programme [20].

4. Decisions and tradeoffs

4.1 Backup-and-restore vs pilot light vs warm standby

Option	Upside	Downside	Revisit when
Backup and restore, single Region	Cheapest. Mostly native AWS features. Hours of RTO is acceptable to most early customers.	Region loss means rebuilding from IaC in another Region with snapshots you haven’t copied there	A customer contract specifies RTO under 4 hours for region loss, or you’re handling funds where downtime costs real money
Pilot light (data replicated, compute switched off)	RPO drops to replication lag. RTO is minutes to an hour.	Every infra change has to work in two Regions. Replication carries corruption across too, so you still need point-in-time backups [3].	You have a dedicated platform team and a real multi-region requirement
Warm standby / active-active	Near-zero RTO	High cost and complexity	Usually only at scale

For most seed to Series A crypto infrastructure companies, the answer is backup and restore done properly: restores verified weekly, immutable copies, a written rebuild order. The expensive work is the verification, not the replication.

4.2 AWS Backup vs native snapshots

	Native (RDS automated backups, ElastiCache snapshots, DLM)	AWS Backup
Cost and complexity	Free or included, one line of IaC per resource	Plans, vaults, selections, IAM
PITR	RDS native, up to 35 days	Also supported. If an RDS instance is in a Backup plan, that plan is used for PITR [7].
Immutability	None. Anyone with `rds:DeleteDBSnapshot` or `DeleteDBInstance` plus “delete automated backups” can destroy them.	Vault Lock (governance or compliance mode) [4] and logically air-gapped vaults [6]
Cross-account / cross-Region copy	Per service, done by hand	Built in
Restore testing	Build it yourself (§3.5)	Managed [5], but no ElastiCache and you write the validation
Coverage	Every service	ElastiCache and Amazon MQ are not covered for restore testing, and MQ has no backup at all

Recommendation: keep native backups as the operational restore path, because they are faster, familiar and already in the runbook. Add AWS Backup with Vault Lock for Tier 0 stores only, as the immutable copy. It’s the cheapest defence against the realistic threat, which is a compromised credential, bad automation or a mistaken bulk delete removing the very snapshots you’re about to restore from.

4.3 Immutability vs cross-account: two separate questions

These two get asked together, but they defend against different threats:

	Defends against	Needs	Typical startup verdict
Immutability (Vault Lock / Object Lock)	Snapshots deleted by a compromised credential, bad automation or human error	Nothing outside the current account	Do it next
Cross-account copy	Loss or compromise of the whole account	An AWS Organization, a second account, cross-account vault policy, and operating the second account	Accepted risk, written down, until an Organization exists for other reasons or a compliance requirement arrives

An account-level event takes the cluster, state bucket, KMS keys and snapshots at once, so the recovery DAG has no starting point; a cross-account snapshot copy is a partial defence against a scenario nothing else covers. Make the decision consistently across the inventory (state bucket and snapshots alike). AWS does recommend cross-account backup because it “helps protect from disaster events that include insider threats or account compromise” [3], so record this as a deliberate risk acceptance with a revisit trigger, not as best practice.

Before you can even plan the cross-account half, check whether an Organization exists. A permissions boundary that denies organizations:* to engineers, which is a good control, also means a normal engineer session can’t answer that question.

4.4 Vault Lock: governance first

Compliance-mode Vault Lock is irreversible by design. After a grace period of at least 72 hours, “the vault and its lock are immutable and cannot be changed or deleted by any user or by AWS”, and any recovery point with “Always” retention is kept forever [4]. Choose the retention as if it were permanent, because in compliance mode it is. Start in governance mode, which a principal with the right IAM permission can reverse. Move to compliance mode only when a regulation requires it or once the retention values have run unchanged for a quarter.

resource "aws_backup_vault" "tier0" {
  name        = "tier0-immutable"
  kms_key_arn = aws_kms_key.backup.arn
}

resource "aws_backup_vault_lock_configuration" "tier0" {
  backup_vault_name  = aws_backup_vault.tier0.name
  min_retention_days = 7
  max_retention_days = 35
  # changeable_for_days = 3   # ← setting this switches to COMPLIANCE mode.
  #                              Leave unset (governance) until retention is proven.
}

resource "aws_backup_plan" "tier0" {
  name = "tier0-daily"
  rule {
    rule_name         = "daily"
    target_vault_name = aws_backup_vault.tier0.name
    schedule          = "cron(0 5 * * ? *)"
    lifecycle { delete_after = 35 }
  }
}

There is one more catch. The backup vault’s KMS key becomes a dependency of every restore from it. The same deny-deletion controls you put on signing keys (§3.7.2) apply here too.

4.5 Signing keys: succession vs escrow vs multi-Region vs imported material

This decision is specific to crypto, and it is one people get wrong in expensive ways.

Option	How it works	Upside	Downside
Succession by contract design	The on-chain identity is a contract (guardian, Safe, owner with `setExecutor`/`transferOwnership`), and the KMS key is only its current signer	Losing a key becomes a rotation. Nothing is stranded. The cleanest answer.	It has to be designed in before deployment. You can’t add it to an EOA-owned proxy afterwards.
Non-exportable KMS, accept I	Single-Region KMS key, prevention controls, documented key-compromise protocol	Strongest custody: key material never leaves the HSM	Region-level KMS loss or accidental deletion is permanent. Fails CCSS’s “backup exists” requirement as written [21].
KMS multi-Region keys	Replica keys with the same key material and key ID in other Regions, usable for signing [12]	Region DR for a signing identity without exporting the key. A replica keeps working “even if its primary key and all related replica keys are disabled”.	Must be chosen at creation. “You cannot convert an existing single-Region key to a multi-Region key” [12]. Each replica is one more principal surface to lock down.
Imported key material (BYOK)	Generate the key in your own HSM or offline ceremony, import it into KMS, keep an escrowed copy	For asymmetric keys, “if you have the original key material, you can create a new KMS key with the same cryptographic properties” after deletion [11]. Meets CCSS backup requirements.	You now run an escrow: split custody, ceremony, tamper evidence, geographic separation. That copy is the most valuable secret you own.
Raw keys in a password manager	A private key stored as a vault field	Easy	No HSM, broad blast radius. And if the password-manager account has one admin, a lost account is potentially unrecoverable.

Recommendations:

For new on-chain identities, put a contract in front of the key. This is where most of the value is. It turns Recoverability I into B.
Decide multi-Region at key creation for Tier 0 signers. The decision costs nothing on day one and can’t be made later.
For raw keys in a password manager, migrate them to KMS rather than adding an escrow. The precondition is that the password manager itself is recoverable: a second admin and an offline emergency kit. Escrow only becomes the right answer if adding a second admin is refused.
Make DR design a launch-blocking review item for any system that will hold an unrecoverable secret. One team’s “pre-production watch list” caught an enclave signer that derived every customer key from one HKDF master seed with no durable home yet. Losing it loses every customer identity, and succession can’t help because customers hold the derived addresses. Escrow added after launch means migrating every key.

For key compromise, as opposed to key loss, the order is containment first (pause services, revoke credentials, per your incident playbook), then succession. CCSS Level II requires a documented Key Compromise Protocol with named roles and backups for each role, and Level III requires that it be tested periodically [21]. NIST SP 800-34’s contingency-plan structure maps directly onto the BG-n template [20].

4.6 Build vs buy for restore checks

Use AWS Backup restore testing if your Tier 0/1 stores are RDS, Aurora, DynamoDB, EBS or S3, and you’re already putting them in Backup plans for immutability. Write the validation Lambda, triggered by the restore-job event, and reuse §3.5.1’s assertions.
Build Step Functions checks if you need ElastiCache, specific per-store assertions (key-class checks, cursor checks), or to validate exactly the native snapshots your runbook would restore from.
Use an in-cluster Kubernetes Job for data that already lives in EKS (vmrestore into an emptyDir, a throwaway server bound to 127.0.0.1, then assert totalSeries > 0). A scratch restore of that data is just a Job, with no managed-service instance to create.

5. Failure modes and field lessons

Every one of these happened on a real platform, in a team that took DR seriously.

5.1 The weekly restore check that never once worked

A team deployed a weekly RDS restore check, and the docs said “confirmed weekly”. While preparing the first manual drill, an engineer found that every run since deployment had failed with AccessDenied. A cloud-provider upgrade had changed a module output from the DB instance identifier to the immutable resource ID. The IAM policy built snapshot ARNs from that output, so the pattern matched no snapshot. The failure alert either hadn’t been wired up yet or lived in a separate pipeline. Nobody had looked at the execution history.

Lessons: a check with no freshness alert is not a check (§3.6). Do the first live drill before you write “verified” anywhere. Pin a test that renders the IAM ARN pattern and compares it to a real snapshot ARN.

5.2 Validation that passes on zero

The same check then “validated” the core database by sampling the first five tables in alphabetical order. All five belonged to a feature that wasn’t in use yet, so the sample was 0 rows. The tables that mattered (quotes, receipts, operations) were never looked at. Always include named critical tables, plus the largest tables by size, and log which tables you sampled. Otherwise a real zero-row result looks the same as a meaningless one.

5.3 Three more from the restore checks

Fail-fast orphans scratch resources (§3.5.2): one store’s reporting exception aborted the whole Map and left a sibling’s scratch instance running, twice. Iterations must never fail, results are aggregated at the end, and a reaper is the backstop.
An empty restore can be correct (§3.5.3): zero keys four runs in a row, because every key had a TTL of 10 minutes or less. Before calling a restore broken, ask what the correct result would be for that store.
A cache that became the system of record (§3.4.2) is the most expensive lesson here. Classify every key prefix. Any store holding non-derivable data gets database-grade durability, or the data moves.

5.4 PersistentVolume traps

The StorageClass says encrypted; the volumes aren’t. StorageClass parameters apply only at provisioning, so older PVs stay unencrypted, and their snapshots inherit that. Check describe-volumes, not the YAML.
“Available” in EC2 does not mean orphaned. EC2 reports a volume available whenever nothing is attached, including one whose PV is still Bound while its pod is unscheduled; PV phases are a separate state machine from EBS attachment [14]. Before deleting any unattached volume, match kubectl get pv -o jsonpath='{..csi.volumeHandle}' against its ID. Without that check, a “clean up unattached volumes” script is a data-loss tool.
Use Retain for anything stateful. Under Delete, removing the PVC removes the underlying asset [14], and repointing a PVC at a restored volume usually means recreating the PVC.
A bootstrap snapshot is not a restore point. A 400 GiB snapshot taken from the dev volume to seed prod looked like a backup; restoring it would have rewound prod to dev’s month-old state.
Placing a restored volume: create it in the AZ the PV’s node affinity pins, reproduce the Name tag the DLM policy selects on, and verify the workload resumed from its cursor, not just that it mounted.
Filter snapshot lookups on a stable tag. A runbook command keyed on the policy-name tag returned nothing after a rename, which mid-incident reads as “no snapshots exist”.

5.5 Cutover traps after a good restore

ignore_changes = [snapshot_identifier] makes “apply with the new snapshot var” plan no changes. You need an explicit -replace on the instance resource. The playbook has to say so, or an engineer will watch a clean plan at 3am and assume the restore applied.
Database roles come from the snapshot. Terraform doesn’t reissue them. Check that each application’s DB user still authenticates.
Not every client reconnects to a new endpoint by itself. One team had source-verified that their postgres.js pool handled an endpoint change. Their Prisma-based services had not been verified, so the playbook says rollout restart them.
Connections that bypass the pooler (a service using the direct endpoint in place of RDS Proxy) need their own manual connection-string update. List them in the inventory’s edge fields.

5.6 Locked down so far you can’t see the keys

A key policy denied kms:GetPublicKey to every principal except the signer, and only from the production NAT IPs. That’s a good security control. But it meant that in a disaster where the production network is gone, nobody can list which addresses they’re trying to save. Engineers and CI runners got explicit denies. Six of thirteen derived addresses were simply unknown.

The tempting fix is to give the ops bastion GetPublicKey. That’s wrong, because in that setup the existing grant came with Sign too, and exempting an extra principal from the signer-only deny is exactly what a SOC 2 reviewer will object to. The right fix is to read each public key once, as its own signer, from inside the network (for example kubectl exec into the running signer pod and make one GetPublicKey call), then write the address into the inventory and into the key’s tags.

5.7 Ownership migration nobody can describe

The key succession playbook covered 5 of 13 production signing keys. For the deployer/owner key, the one with the largest blast radius (owner of every proxy, part of every CREATE2 initcode), no repository the infra team could see documented how to transfer ownership. The playbook could only say “escalate to the contract owners.” Write that down honestly, and treat it as a Tier 0 finding. Then go and find the people who know.

5.8 Docs more confident than reality

An architecture document listed a monthly RDS drill, a cross-region replica for the audit log, and RTOs that matched nothing measured. None of it was true. The team deleted the section and replaced it with links, rather than “correcting” it in place, and made a rule for every DR document: state only what is true now, and replace a claim when it changes instead of appending a correction next to it. The DR router keeps a “never” column, and empty playbook cells are labelled “a gap, not a decision”.

5.9 The precondition that turned out false

A secrets-loss playbook listed “at least one other password-manager admin” as a precondition for account recovery. When someone checked, the account had exactly one admin, and nobody had confirmed an offline emergency kit existed. So recovery of the vault holding the raw production keys depended on one person. The fix took two minutes. It was found only because someone wrote the precondition down and then verified it. Every precondition in every playbook should be verified this way.

6. Best-practice checklist

Day 1 (seed, one engineer-week)

Inventory every stateful asset via an IaC grep plus a live sweep of every Region, with tier, recoverability (R/B/I) and owner, even if the owner is “unassigned”.
RDS: backup_retention_period ≥ 14 (30 for Tier 0), deletion_protection, final snapshot, enforced in the module.
Versioning on the Terraform state bucket and every bucket you’d want data back from.
Signing keys: prevent_destroy, 30-day deletion window, CI-role Deny on ScheduleKeyDeletion/DisableKey/PutKeyPolicy, and a SEV1 alert on ScheduleKeyDeletion.
Record each signing key’s derived address and its provenance, and tag the key with the address.
Classify every cache key prefix. Move non-derivable data to a durable store.
One manual RDS restore into a scratch instance, timed and written up as the first drill log.
A DR router page listing each scenario with its playbook or “not yet written”.

First year (Series A)

Automated weekly restore-and-validate for every Tier 0/1 store, with Step Functions or AWS Backup restore testing, named-table assertions, a reaper, and scoped IAM.
Freshness alerts (“no success in 8 days”, including absent()) for every restore check and every backup producer (DLM, Backup jobs, definitions export, vmbackup, audit export).
An external heartbeat on the alerting system itself.
AWS Backup plus Vault Lock (governance mode) for Tier 0 RDS.
Broker definitions exported daily to versioned, KMS-encrypted S3. Consumers self-declare their topology.
A BG playbook for each Tier 0 store and for key succession, each with an irreversibility gate. One live drill per playbook, propagated.
A recovery DAG with known cycles listed. A mirror of the infra repo outside the git host.
Decommissioning procedure for signing services: forensic capture → sweep → disable → observe → delete.
Password manager: at least two admins and a verified offline recovery kit.
New on-chain identities behind a rotatable contract. New Tier 0 KMS signers created multi-Region.

At scale

AWS Organization, a dedicated backup account, cross-account copies or a logically air-gapped vault [6].
Compliance-mode Vault Lock where regulation requires it, with retention you have already run for months.
Playbooks for account compromise, region loss and mass secret rotation, each tabletop-tested.
Restore tests on random recovery points, not only the latest [5].
Game days with injected faults. Drill results reported to leadership like any other SLO.
A documented, periodically tested Key Compromise Protocol meeting CCSS Level III [21].

7. Maturity roadmap

Level	What’s true	Typical evidence	Next step
0: Backups exist	Default RDS backups, maybe ElastiCache snapshots. Nobody has restored one.	“Backup succeeded” in the console	Inventory, plus one manual restore
1: Inventoried	Every asset has a tier, recoverability class and owner field. The irreplaceable assets are named.	Inventory file. Signing-key table with addresses.	Automate a restore check for the Tier 0 DB
2: Verified	Weekly automated restore-and-validate for Tier 0/1. RTOs measured, not estimated.	Execution history. Drill logs. Measured RTO column.	Freshness alerts. Immutability.
3: Monitored and immutable	Freshness alerts on every producer and check. Tier 0 copies in a locked vault. Heartbeat on alerting.	Alert rules. Vault Lock config.	Playbooks with gates. Drill each one.
4: Rehearsed	BG playbooks for every Tier 0 scenario, each drilled or tabletop-tested. Decommissioning procedure used at least once.	Dated drill logs with propagation ticked	Account-level isolation
5: Isolated	Separate backup account or air-gapped vault. Account-compromise and region-loss playbooks. Key succession designed into every contract.	Org structure. Cross-account restore drill.	Continuous: random-point tests, game days

Most well-run startups sit at Level 2 or 3, with honest gaps written down. The aim isn’t Level 5 at Series A. The aim is that your documents describe your actual level accurately, so a partner, an auditor or an engineer at 3am can rely on what they read.

8. References

AWS Well-Architected Framework, Reliability Pillar. Back up data (REL09-BP01 to BP04). https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/back-up-data.html
AWS Well-Architected Framework, Reliability Pillar. Plan for Disaster Recovery (DR) (REL13-BP01 to BP05). https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/plan-for-disaster-recovery-dr.html
AWS Whitepaper. Disaster Recovery of Workloads on AWS: Disaster recovery options in the cloud. https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html
AWS Backup Developer Guide. AWS Backup Vault Lock. https://docs.aws.amazon.com/aws-backup/latest/devguide/vault-lock.html
AWS Backup Developer Guide. Restore testing. https://docs.aws.amazon.com/aws-backup/latest/devguide/restore-testing.html
AWS Backup Developer Guide. Logically air-gapped vault. https://docs.aws.amazon.com/aws-backup/latest/devguide/logicallyairgappedvault.html
Amazon RDS User Guide. Restoring a DB instance to a specified time. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html
Amazon ElastiCache User Guide. Snapshot and restore. https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/backups.html
Amazon EBS User Guide. Automate backups with Amazon Data Lifecycle Manager. https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-lifecycle.html
Amazon S3 User Guide. Locking objects with Object Lock. https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html
AWS KMS Developer Guide. Delete an AWS KMS key. https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html
AWS KMS Developer Guide. Multi-Region keys in AWS KMS. https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html
AWS Step Functions Developer Guide. Using Map state in Inline mode. https://docs.aws.amazon.com/step-functions/latest/dg/state-map-inline.html
Kubernetes Documentation. Persistent Volumes. https://kubernetes.io/docs/concepts/storage/persistent-volumes/
RabbitMQ Documentation. Definitions (schema export and import). https://www.rabbitmq.com/docs/definitions
VictoriaMetrics Documentation. vmbackup. https://docs.victoriametrics.com/victoriametrics/vmbackup/
HashiCorp Terraform Documentation. Backend type: s3. https://developer.hashicorp.com/terraform/language/backend/s3
Beyer, Jones, Petoff, Murphy (eds.). Site Reliability Engineering, Chapter 26: “Data Integrity: What You Read Is What You Wrote.” Google. https://sre.google/sre-book/data-integrity/
Site Reliability Engineering, Chapter 28: “Accelerating SREs to On-Call and Beyond” (Disaster Role Playing / Wheel of Misfortune). Google. https://sre.google/sre-book/accelerating-sre-on-call/
NIST SP 800-34 Rev. 1. Contingency Planning Guide for Federal Information Systems. https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final
CryptoCurrency Certification Consortium (C4). CryptoCurrency Security Standard (CCSS): Details (Aspects 1.03 Key Material Storage, 1.05 Key Compromise Policy). https://cryptoconsortium.org/cryptocurrency-security-standard-documentation/details/