Building on AWS · guide 07 of 13

Secrets, Identity and Signing Keys for Blockchain Startups on AWS

7722 words · about 39 minutes · all guides

Abstract. For most SaaS companies a leaked secret means a data breach. For a crypto infrastructure company, it can mean funds leaving an address within seconds, irreversibly, with no chargeback and no support ticket to file. This article is for senior DevOps, SRE and platform engineers at startups that hold transaction-signing keys on AWS: relayers, solvers, bridges, custodial and semi-custodial wallets, account-abstraction paymasters, fee collectors. It covers using AWS KMS asymmetric keys (ECC_SECG_P256K1, ECC_NIST_EDWARDS25519) as signers, and why the key policy AWS gives you by default is not a signing boundary. It shows how to write principal and network restrictions that hold up, how to stop a key from being destroyed by accident, and how to plan succession for an on-chain identity whose address can never be rotated. It then covers the rest of the secrets estate: where raw secrets live (1Password Connect, External Secrets, Secrets Manager, SSM), how much of it CI can see, what ends up in Terraform state, human IAM with MFA, break-glass and offboarding. Finally it looks at Nitro Enclave signers with attestation, for when KMS stops being enough. The patterns and failure modes come from a production platform that runs about a dozen fund-moving KMS keys across two environments. The details have been generalised.

1. The problem space

1.1 Why signing keys are different from every other secret

Most secrets can be rotated. When a database password leaks, you issue a new one, roll the pods and write a post-mortem. Signing keys for on-chain identities break this model in three ways.

The address is the identity, and you can’t rotate it. An EVM address is the last 20 bytes of the Keccak-256 hash of the public key. Contracts store it as owner, executor or guardian. Allowlists, multisig signer sets, counterparties and CREATE2 initcode embed it, and so do your own database rows. “Rotating” the key really means a migration: you deploy a new key, then repoint every on-chain and off-chain reference to it while the old key still works. If there is no setter for a reference, it cannot be migrated.
A compromise is final. Nobody can reverse a transaction once it is broadcast. Detection after the fact helps you write the incident report, but it does not get the funds back. So prevention controls (who can call Sign) matter far more than detection controls (who did call Sign).
Destroying the key is also final. Deleting a symmetric key leaves you with ciphertext you can no longer read. Deleting a signing key strands the identity itself: whatever that address owns, governs or holds can never be moved again. AWS’s own deletion guidance is written mostly around encryption keys [4], but the consequence for a signer is worse.

1.2 The forces at play in a startup

Small team, big blast radius. Often one or two people hold AWS administrator access. “Admin anywhere = admin everywhere” is the default in a single AWS account, and it remains the root cause of most key-exposure findings long after the team grows.
Speed. Developers need to sign from laptops against testnets. Services need new keys for new chains every few weeks. Every control has to survive a “just give me access to test this” request.
Partner and auditor pressure. Exchanges, custodians and enterprise customers increasingly ask for SOC 2 evidence and for alignment with the CryptoCurrency Security Standard (CCSS) [9]. CCSS has explicit aspects for key generation, key storage, key usage and a Key Compromise Policy. CCSS reserves its highest compliance level for organisations that have a documented key-compromise protocol. “We use KMS” is a good answer to key storage. It is not an answer to key usage or key compromise.
Cost. A KMS key costs about $1/month plus per-request charges. CloudHSM costs thousands of dollars a month per cluster. Nitro Enclaves cost an EC2 instance plus a large engineering effort. Most startups should be on KMS, so the engineering effort should go into using KMS correctly.

1.3 What “correct” means

The rest of the article argues for this target posture:

Every fund-moving key is a non-exportable KMS key, owned by exactly one environment’s state. It can be used for Sign only by named workload roles, only from known network paths. No human, no CI role and no account administrator can sign with it, except through a documented, audited break-glass path. It cannot be deleted by a Terraform refactor. For every on-chain reference to it, a migration procedure has been written and rehearsed before anyone needs it.

2. Reference design

2.1 Architecture

Where signing keys live and who can reach them: workload identity, the KMS key, and the network path that reaches it. full size ↗

Today the design runs in a single account, held together with explicit Denies. Section 4.1 covers that tradeoff. The key policy shape is the same in both cases. What changes is how much damage an administrator principal in the wrong environment can do.

2.2 The key policy: why the default is not a boundary

When you create a KMS key through the API and pass no policy, AWS applies exactly one statement [1]:

{
  "Sid": "Enable IAM User Permissions",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": "kms:*",
  "Resource": "*"
}

This statement does not mean “only the root user”. It delegates key authority to IAM: any principal in the account whose identity policy allows kms:Sign on * (or on this key) can sign. In a typical startup account, that includes:

every human with AdministratorAccess;
every CI deploy role with AdministratorAccess, which is common because writing least-privilege deploy roles is expensive;
any role carrying a wildcard KMS statement that someone added “temporarily” to debug something else.

AWS keeps this statement for a good reason: it stops the key from becoming unmanageable if the one principal named in a stricter policy is deleted [1]. You do not want to remove it, because that is how teams end up opening an AWS Support case to regain control of their own key. You want to keep it for management and override it for use.

The tool for this is the explicit Deny. In IAM policy evaluation, an explicit Deny in any applicable policy beats every Allow in every other policy. A Deny in the key policy applies to every caller, whatever their identity policies grant today or are granted next year.

locals {
  signing_actions = ["kms:Sign", "kms:GetPublicKey"]

  key_policy = jsonencode({
    Version = "2012-10-17"
    Statement = concat(
      [{
        # Keep: lets IAM manage the key and stops lockout.
        Sid       = "EnableIAMPolicies"
        Effect    = "Allow"
        Principal = { AWS = "arn:aws:iam::${local.account_id}:root" }
        Action    = "kms:*"
        Resource  = "*"
      }],

      # 1. Positive principal scoping, written as a Deny on everyone else.
      length(var.allowed_principal_arns) > 0 ? [{
        Sid       = "DenySigningByUnapprovedPrincipals"
        Effect    = "Deny"
        Principal = { AWS = "*" }
        Action    = local.signing_actions
        Resource  = "*"
        Condition = { ArnNotEquals = { "aws:PrincipalArn" = var.allowed_principal_arns } }
      }] : [],

      # 2. Network scoping. A caller escapes this Deny by meeting ANY one exemption.
      length(var.allowed_source_ips) > 0 ? [{
        Sid       = "DenySigningOutsideKnownNetworkPaths"
        Effect    = "Deny"
        Principal = { AWS = "*" }
        Action    = local.signing_actions
        Resource  = "*"
        Condition = merge(
          { NotIpAddress = { "aws:SourceIp" = var.allowed_source_ips } },
          length(var.trusted_vpce_ids) > 0 ?
            { StringNotEquals = { "aws:sourceVpce" = var.trusted_vpce_ids } } : {},
          length(var.ip_exempt_principal_arns) > 0 ?
            { ArnNotEquals = { "aws:PrincipalArn" = var.ip_exempt_principal_arns } } : {}
        )
      }] : [],

      # 3. Retired key: nobody signs, no conditions, nothing to reason about.
      var.deny_all_signing ? [{
        Sid       = "DenyAllSigning"
        Effect    = "Deny"
        Principal = { AWS = "*" }
        Action    = local.signing_actions
        Resource  = "*"
      }] : []
    )
  })
}

These four statements each encode a lesson the hard way.

Principal scoping is a Deny on the complement, not a narrower Allow. You could replace the root statement with “Allow kms:Sign to role X”. But then IAM can no longer manage the key, and a stray identity policy elsewhere still can’t be overridden by anything. A Deny on ArnNotEquals aws:PrincipalArn covers only Sign/GetPublicKey. GetKeyPolicy, PutKeyPolicy and DescribeKey keep working, so Terraform still manages the key and nobody can lock the account out of its own policy. Note that for an assumed-role session, aws:PrincipalArn resolves to the role ARN (arn:aws:iam::<acct>:role/<name>), not the session ARN. Name IRSA and Pod Identity roles by their plain role ARN.

The network Deny must use Principal: "*". One common first version scopes the IP Deny to the key’s own IAM user. Deny statements apply only to the principals they name. So a CI role, or any other identity with a wildcard KMS grant, can then sign from anywhere, and the IP check becomes decoration.

aws:SourceIp does not work through a VPC endpoint. AWS documents this directly: aws:SourceIp “is not effective when the request comes from an Amazon VPC endpoint”. Use aws:sourceVpce or aws:sourceVpc instead [2]. A call that goes through an interface endpoint carries the endpoint ENI’s private IP, never your NAT EIP. One team learned this in production. Someone enabled private DNS on the region’s KMS interface endpoint, a networking change that looked unrelated, and every kms:Sign from the signing service silently started going through the endpoint. The key policy’s NAT-IP-only Deny then refused every signature, and nobody had touched the key. The fix is the third clause above: treat a trusted endpoint ID as an alternative network proof. And whenever you add or change a KMS endpoint, check every key policy that conditions on source IP.

“Deny everyone” must not be expressed as an empty allowlist. If the module skips the principal Deny when the list is empty (as most modules do, so that unscoped keys keep working), then allowed_principal_arns = [] means “no restriction”, not “nobody”. Retiring a key by deleting its only consumer from the list therefore opens it to every administrator. Use a separate, unconditional DenyAllSigning statement, so that no condition-operator edge case can change what it does.

Two further conditions are worth adding to high-value keys:

kms:SigningAlgorithm [19] can pin a key to a single algorithm (for example ECDSA_SHA_256 for EVM signing over a precomputed digest). It narrows what a stolen role session can do with the key, and it documents intent.
kms:RecipientAttestation:* for enclave-held material. Section 2.6 covers this.

2.3 Lifecycle: make destruction a deliberate act

resource "aws_kms_key" "signer" {
  description              = "relayer-prod — owns relayer EOA on all EVM chains"
  key_usage                = "SIGN_VERIFY"
  customer_master_key_spec = "ECC_SECG_P256K1"   # immutable: changing it = new key = new address
  deletion_window_in_days  = 30                  # max; default is also 30
  policy                   = local.key_policy

  lifecycle {
    prevent_destroy = true
  }
}

prevent_destroy stops terraform plan from proposing a destroy. A refactor that renames a module, moves a key between state files or deletes a “dead-looking” block will fail at plan time and not at apply time. To decommission a key on purpose, someone has to edit the lifecycle block in a reviewed diff. One independent review of a mature platform found that its signing-key module lacked prevent_destroy, even though its DNSSEC key had it. Add it to the module so that every key gets it, rather than relying on each caller to remember.
key_spec is immutable. Changing it, even by “correcting” a default, plans a replacement: a new key with a new address. prevent_destroy will refuse the plan, and you want it to.
The deletion window (7–30 days, default 30, and the real time can be up to 24 hours longer [4]) is your last chance to undo. A key pending deletion cannot sign, so the window is also a soft kill switch. Set an alarm on any use attempt during the window, and alert on ScheduleKeyDeletion in CloudTrail itself.
Deny the destructive actions on automation roles. If deploy roles keep broad rights (section 4.2), put an explicit Deny for kms:ScheduleKeyDeletion, kms:DisableKey and kms:PutKeyPolicy on them. Carve exceptions only for the environment that owns a given key.
Know the chicken-and-egg problem. A guardrail that denies PutKeyPolicy on keys outside an allowlist, where the allowlist is resolved by alias lookup, will block the first apply of a new key. The key has no alias yet, so it can’t be on the list, and the aws_kms_key_policy resource then fails. The workaround is to send the policy with CreateKey (the policy argument on aws_kms_key), together with bypass_policy_lockout_safety_check. AWS documents the bypass for exactly this case: a caller that is deliberately barred from PutKeyPolicy on the new key. The root statement keeps the key manageable. Add the alias to the allowlist in a follow-up merge.

2.4 Workload identity: who calls `Sign`

Use IRSA or EKS Pod Identity, never static access keys. Each signing service gets its own role, bound to one Kubernetes ServiceAccount in one namespace [17]. That role ARN goes into allowed_principal_arns. Per-service roles also give you per-service CloudTrail attribution.
Retiring a legacy IAM user is a two-step operation. If the key was first reached through an IAM user whose access key was created by hand, Terraform will fail to delete the user (DeleteConflict) because the access key is not in state. Worse, that failed destroy then re-fails on every later apply of the whole environment, so everyone who merges after you inherits a red pipeline. Declare the credential deletion instead (force_destroy = true on aws_iam_user). Land it in its own apply first: force_destroy is read from state at destroy time, and a count = 0 instance has no config left to read it from. Setting both flags in one merge changes nothing.
Derive the address from inside the trust boundary. For a key that only its workload role can call GetPublicKey on, nobody else can read the address from a laptop. Read it once from inside the running workload (kubectl exec and an SDK call) and record it in the key’s documentation and your asset inventory. This is the intended outcome, not a nuisance: in one team’s words, “a prod executor key that a laptop can sign with is a prod executor key that a laptop can trade with.”
Get the signature handling right. KMS returns a DER-encoded ECDSA signature with no recovery ID. For Ethereum you must parse r and s and normalise s to the lower half of the curve order (EIP-2 made signatures with s > secp256k1n/2 invalid [16]). You then derive v by trying both recovery IDs against the known address. AWS’s reference series walks through this [6]. Use a maintained library. A homegrown version tends to work on 50% of signatures in tests and fail on the other 50% in production.

2.5 Raw secrets: what is not a KMS key

Not every “key” belongs in KMS. The deciding question is where the cryptography runs:

Consumer does…	Correct store	Why
Remote `kms:Sign` / `kms:GenerateMac` over an API	KMS key (asymmetric `SIGN_VERIFY`, or HMAC `GENERATE_VERIFY_MAC`)	Key material never leaves the HSM boundary
Local `crypto.createHmac(secret, …)`, JWT HS256, webhook signing	A raw secret in a secret store	A KMS HMAC key can never export its material, so it cannot produce the string your code needs
Third-party API key or DB password	Secret store, with rotation where possible	Long-lived, replaceable

One team built Terraform for a pair of KMS HMAC keys and IAM users before anyone read the consumer code, which turned out to compute HMAC-SHA256 locally over an env var. The PR was closed. The lesson is to read the consuming code path before choosing a primitive.

There is a second lesson about shared symmetric signing authority. If several services share one HS256 JWT secret, any holder can forge any other holder’s identity. Keep that set frozen, give each new integration its own credential, and treat “add service X to the shared JWT secret” as a security review item, not a config change.

2.6 When KMS is not enough: enclave signers with attestation

KMS solves “the key is never exposed”. It does not solve “the key is used only for transactions that pass policy”. Any principal allowed to call Sign can sign any digest. If you need signing to depend on business rules (spend limits, allowlisted destinations, recovery flows that check a guardian proof), you either trust the application that calls KMS, or you move the policy and the key into a Trusted Execution Environment (TEE).

AWS Nitro Enclaves provide an isolated VM with no persistent storage, no interactive access and no external networking. The only channel is vsock to the parent instance. KMS supports enclave attestation natively: the enclave attaches a signed attestation document to Decrypt, GenerateDataKey or GenerateRandom, and the key policy can require specific measurements via kms:RecipientAttestation:ImageSha384 and kms:RecipientAttestation:PCR<n> [3][5]:

{
  "Sid": "DecryptOnlyInsideApprovedEnclave",
  "Effect": "Deny",
  "Principal": { "AWS": "*" },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringNotEqualsIgnoreCase": {
      "kms:RecipientAttestation:PCR0": ["<current-pcr0>", "<next-pcr0>"]
    }
  }
}

A design one team built for a recovery-signing fleet shows what a serious TEE signer actually involves:

Two enclaves per host. A policy enclave evaluates requests. A signer enclave holds key material in RAM only, and the key is replicated between hosts in different AZs by mutually attested, exact-measurement gossip. No KMS key holds the signing key. KMS only protects the policy enclave’s configuration, through attestation-gated Decrypt.
An approved-PCR0 list of at most two entries (current and next). Rolling out a policy release takes a fixed sequence of reviewed changes: add the new PCR0, roll out, then remove the old one. A new KMS key starts with an empty list, so it decrypts for no enclave at all. The policy also explicitly denies ReEncrypt* and GenerateDataKey*, which would otherwise be alternate paths to plaintext.
Release signing with an asymmetric KMS key (ECC_NIST_P256) and offline verification against a pinned public key. The first design used keyless Sigstore signing, which made every relay restart and signer start depend at runtime on public transparency-log infrastructure. They replaced it. Only two publisher roles can sign, through an explicit Deny on every other principal, administrators included.
Hosts that are hard to kill by accident. prevent_destroy, disable_api_termination, disable_api_stop, automatic recovery disabled, no SSH (sshd masked, EC2 Instance Connect removed), and deploy-role Denies on Stop/Reboot/Terminate, SSM SendCommand/StartSession, CreateImage and metadata changes on instances tagged Custody=enclave-ram. A CI plan guard rejects any update, replacement, move or drift of custody resources.

The review of that design is just as instructive as the design. Expect these when you build a TEE signer:

The 16 KB EC2 user-data limit is real. Inlining verifier scripts pushed gzipped user data to about 17.5 KB, and instance creation failed. They added a CI test with a 15 KB budget.
ignore_changes plus prevent_destroy freezes bugs in place. A host-side release verifier that predated a manifest format change rejected every new release, and hosts launched with it could never be repaired through Terraform. Anything immutable after launch must be correct at launch, so test it against current artefacts before the first host exists.
RAM-only custody has a long recovery tail. A rebooted signer re-joins custody, but by design it refuses policy sessions for 48 hours, and during that window routing operations for the whole fleet stall. For a recovery system that is an acceptable cost. For a hot payment path it is not.
Expect the vsock addressing to need hands-on proof (parent CID 2 vs 3, VMADDR_CID_ANY listeners). Nothing about it is proven until it runs on real Nitro hardware.

The honest summary: a TEE signer is a quarter or more of focused engineering for a small team, and it is the right answer only when you need policy-enforced signing or non-AWS-trusting custody. For “a backend service signs transactions”, well-scoped KMS is the stage-appropriate answer.

3. The rest of the secrets estate

3.1 Choosing a store

Store	Strengths	Weaknesses	Use for
1Password + Connect operator [10]	Humans already live there; version history; non-engineers can rotate a vendor key; vault-scoped tokens	Another control plane to run and secure; Connect credential bootstrap is fiddly; 60s polling	Human-managed third-party API keys, shared app config secrets
External Secrets Operator [11] (backed by Secrets Manager / SSM)	AWS-native IAM, CloudTrail on every read, many backends	Values reach Kubernetes Secrets just like any operator; one more controller	AWS-generated credentials, a multi-backend estate
AWS Secrets Manager [12]	Rotation Lambdas, RDS integration, resource policies, ~$0.40/secret/month	Cost at scale; the Terraform-managed `secret_version` lands in state	Machine credentials, bootstrap credentials for other systems
SSM Parameter Store (SecureString)	Free standard tier, hierarchical paths	No native rotation, lower throughput, 4 KB/8 KB limits	Non-secret config, low-value secrets, cross-stack lookups

Many teams end up with a hybrid: 1Password for secrets that humans manage, Secrets Manager for secrets that machines generate, and KMS for anything that signs. Whichever store you pick, the values end up as a Kubernetes Secret unless you mount them through the Secrets Store CSI driver. So follow the Kubernetes guidance: encryption at rest, list/watch on Secrets only for system components, and remember that anyone who can create a Pod in a namespace can read that namespace’s Secrets [13].

3.2 1Password Connect field notes

The 1password-credentials.json for a Connect server is issued exactly once and can never be downloaded again. op connect server supports only create, get, edit, delete and list. If you lose the file before storing it durably, you have to delete the server, which revokes every token it issued. Store it in its destination (Secrets Manager) before you clean up the local copy.
Scope each Connect server to one vault, and give it a read-only token. A shared-tooling cluster that needs a few prod values should get a server scoped to read the Prod vault. It should not reuse a read-write credential that covers the whole vault.
Split monolithic items. One monolithic “prod-secrets” item holding every service’s secrets means that editing any field restarts every consumer, so teams switch operator auto-restart off, and then rotations silently don’t take effect. Split into per-service items plus one common item.
Moving a reference between items takes two phases. If ArgoCD syncs application references from HEAD the moment a PR merges, but the new item’s Kubernetes Secret is created by a Terraform apply that runs minutes later, then a single-PR “create item and flip references” leaves pods pointing at a Secret that doesn’t exist yet. First create the items (PR 1), confirm the Secrets exist in the cluster, then flip the references (PR 2).
Read values from the owning service’s item; don’t copy them. If service B validates a token that service A issues, B should read A’s item and not hold a copy. Copies drift, and drift in a shared secret shows up as an authentication outage.

3.3 Scoping CI’s view of secrets

CI is the largest secret-exposure surface most startups have, and it grows quietly.

A PR plan job can run arbitrary code. A Terraform plan executes providers, external data sources and, in some setups, a local-exec in a data source. So any role a pull_request-triggered workflow can assume is reachable by anyone who can open a PR (on a public repo, also from a fork if you misconfigure the trigger). One independent review found that a typical “infra-plan” role had all of the following:

ReadOnlyAccess, plus secretsmanager:GetSecretValue and kms:Decrypt on *;
SSM port-forward rights to a prod bastion;
read access to the state bucket and its CMK;
reachability of a Secrets Manager secret holding a still-funded wallet key for a decommissioned Lambda.

On top of that, the PR job received an ArgoCD SSH key, OAuth client secrets, Slack tokens and, on some plans, a 1Password service-account token for prod. An earlier ticket covering exactly this exposure had been closed as “Done” after a docs-only change. Ticket status is not verification: grep the IAM policy.

The remediation pattern follows OWASP’s CI/CD guidance (scope pipeline credentials to only the secrets they need, and log their use) [21]:

Scope GetSecretValue to an explicit list of secret ARNs, and grant kms:Decrypt only through kms:ViaService = secretsmanager.<region>.amazonaws.com plus the state CMK. The cost is that any new aws_secretsmanager_secret_version data source must add its secret name to the list in the same PR, otherwise PR plans (and the scheduled drift plan, which reads managed versions on refresh) fail with AccessDenied. That friction is the point.
Split the role. A PR role should get no secrets and dev-only state. A main-only role, trusted on repo:<org>/<repo>:ref:refs/heads/main [8], plans prod.
Scope GitHub secrets to the matrix jobs that need them. A GitHub secret referenced in a step’s env: block is materialised in every matrix job that runs that step. Gate it on the environment:
```
env:
  OP_CONNECT_TOKEN: ${{ (matrix.environment == 'prod' || matrix.environment == 'prod-apps') && secrets.OP_CONNECT_TOKEN_PROD || '' }}
```
Prefer Secrets Manager over GitHub secrets for bootstrap credentials. A GitHub secret becomes an env var that any step can read, and its reads are not audited. A Secrets Manager read is a CloudTrail event, tied to a role, that you can alert on.
Sweep any funded wallet key before deleting its secret. Removing the resource from state (removed { lifecycle { destroy = false } }) does nothing about old state versions in a versioned bucket, which still contain the value.

3.4 Secrets in Terraform state

Terraform stores sensitive values in plain text in state and plan files. “Anyone who can access those files can access your sensitive values” [7]. Common leaks:

Database outputs that embed the master password in a connection URI, consumed by other stacks through terraform_remote_state. Every reader of that state now holds the master credential. Use per-application DB roles, and publish only endpoints.
aws_iam_access_key resources whose secret lands in state and is exported as an output.
Saved plan files (terraform plan -out) that contain the same values. Never upload them as CI artefacts without encryption and a short retention period.
Reverse edges in the remote-state graph, where prod reads shared and shared reads prod. They cause cycles, and they force a from-scratch rebuild to be done as hand-staged -target applies.

Mitigations, in order of payoff: (1) stop exporting secrets as outputs, and replace cross-stack secret reads with SSM parameter or Secrets Manager lookups at runtime; (2) adopt ephemeral resources and write-only arguments (Terraform 1.10+/1.11+), which keep values out of state and plans entirely [7]; (3) encrypt the state bucket with a dedicated CMK whose key policy names the roles allowed to decrypt, and enable versioning with a lifecycle policy, because old versions keep old secrets.

3.5 Human identity

MFA enforcement with a twist. AWS’s sample self-service policy denies everything except a short list of IAM actions unless aws:MultiFactorAuthPresent is true, and the list includes iam:CreateVirtualMFADevice and iam:EnableMFADevice, so that users can enrol themselves [14]. The problem: an attacker holding a stolen long-term access key, and no MFA device registered for that user, can then enrol their own device and mint an MFA-present session. A stricter variant leaves those two actions out of the exemption. Enrolment then becomes admin-assisted:

An already-MFA’d admin creates the virtual device and gives the QR code to the user out of band (never over chat, because the QR code encodes the raw TOTP seed).
The user reads back two consecutive codes over voice or screen-share.
The admin activates the device and shreds the QR file.

The first admin has to be bootstrapped through the root user. Day to day, users run sts get-session-token --serial-number … --token-code … through a small wrapper script that writes a default-mfa profile with 12-hour credentials. Long-term keys alone get 403 everywhere.

IAM Identity Center is the better end state. You get short-lived credentials by default, central MFA, and offboarding happens in one place (your IdP). Remaining on IAM users is defensible at seed stage. It is a finding at Series A.

Permission boundaries on humans (for example, “you cannot modify credentials or policies of anyone but yourself”) have a side effect: offboarding must go through Terraform, because a hand-run DeleteUser gets AccessDenied. That is a feature, as long as the runbook says so.

3.6 Offboarding

A generalised checklist from a team that learned each line the hard way:

Audit before revoking. list-attached-user-policies, list-user-policies, list-access-keys and list-groups-for-user. Direct grants are common and often not in Terraform. In one audit, five humans had AdministratorAccess attached directly to the user, so removing them from the admin group would not have removed their admin rights.
Revoke and delete in two separate PRs. First move the user to an “offboarding” list, which removes groups, VPN peers, bastion membership and EKS access entries. Delete the user only after that apply lands, especially if any other stack looks the user up with a data "aws_iam_user".
Remove the things Terraform doesn’t fully own. For example, WireGuard peers on the live gateway (usually two per person, laptop and phone), hard-coded approver lists in workflow files, and anything else grep -rni <name> finds.
Check signing keys specifically. Did this person have a human exemption on any key’s IP Deny, or sign locally with a dev key? If a prod key’s principal list ever included their user, treat that as a key-compromise review (section 5) and not just an access removal.
Vendor consoles last but one; the identity provider last. Earlier steps need the account to still be alive.

3.7 Break-glass

Break-glass means emergency access, used rarely, alarmed loudly and drilled [15]. For key management specifically:

The root user is the ultimate break-glass for a KMS key whose policy locked everyone else out. Keep the root credentials on a hardware MFA device, store them in a sealed and audited location, and alert on any root sign-in.
A break-glass signing role, if you need one, should be named in allowed_principal_arns but be assumable only with MFA, by two named humans, with a CloudTrail alert on AssumeRole. Most teams don’t need one. For fund recovery, the safer path is usually an on-chain owner multisig, not a human who can call kms:Sign on the hot key.
Write the break-glass playbooks before you need them, one per scenario: a key compromised, a key lost or scheduled for deletion, the secret store unavailable, the GitHub org lost. Each one needs decision gates, irreversibility warnings and a verification checklist. Record “Last drilled: never” honestly until you have drilled it.

4. Decisions and tradeoffs

4.1 One account with Deny patches vs per-environment accounts

	Single account + explicit Denies	Per-env accounts under AWS Organizations + SCPs
Setup cost	Low	Medium (Control Tower / landing zone, SSO, cross-account CI roles)
Admin-anywhere risk	High; mitigated statement by statement	Structurally removed: a dev admin has no rights in prod
Key isolation	Key policy Denies	Account boundary + key policy + SCP
Ops friction	Low	Cross-account networking, logging, billing views

Recommendation: start single-account if you must, but put Principal: "*" Denies on every fund-moving key from day one. Plan the split when either (a) you have more than about three people with admin, or (b) a partner or auditor asks for environment separation. The AWS Security Reference Architecture describes the target account layout (security tooling, log archive, workloads split by environment) [18]. In one independent review, the single account was the root cause of the critical finding and most of the high ones. Denies are compensating controls; accounts are the actual fix.

Signal to revisit: you find yourself writing a Deny for a non-prod role that names a prod resource.

4.2 Least-privilege deploy roles vs admin plus guardrail Denies

Writing a real least-privilege policy for a Terraform deploy role means deriving permissions from plan output across every resource type you use. That is weeks of work, and it breaks every time you add a resource type. The pragmatic middle ground is AdministratorAccess plus an inline guardrail policy that explicitly denies the irreversible and cross-environment actions:

creating IAM users, access keys or login profiles;
editing the deploy roles’ own trust policies or the OIDC provider;
kms:ScheduleKeyDeletion, DisableKey and PutKeyPolicy outside the environment’s own keys;
organizations:*;
cloudtrail:StopLogging/DeleteTrail and Config recorder teardown;
and kms:Sign on every role that has no business signing.

That last item gets missed. One review found guardrails that denied key deletion and policy changes, but not signing. The un-gated dev deploy role, being an account-wide admin, could therefore sign with the most privileged prod key, a protocol-owner key that had no principal restriction because developers signed with it locally.

Signal to revisit: a guardrail Deny gets a third carve-out, or an auditor asks for SoD evidence. Workflow-level approval gates (for example, an issue comment from an approver list) give you an audit trail and a deliberate act, but anyone who can edit the workflow can remove them. They are not separation of duties.

4.3 KMS vs CloudHSM vs MPC vs TEE

Option	Key exposure	Policy enforcement	Cost	When
KMS (FIPS 140-validated HSMs, multi-tenant)	Never leaves the HSM	Caller-side only (IAM + key policy)	~$1/key/mo + requests	Default for service signers
CloudHSM	Single-tenant HSM, you own the crypto officers	Caller-side	~$1.5–2k/HSM/mo, need ≥2	Regulatory need for single tenancy
MPC / threshold vendor	Key never assembled	Vendor policy engine, quorum	Vendor pricing	Treasury, custody, cross-org approval
Nitro Enclave signer	In enclave RAM only	Enforced in measured code	EC2 + heavy engineering	Policy-enforced signing, attested custody
Smart-contract multisig / account abstraction	Many keys, each low-value	On-chain	Gas	Ownership, upgrades, treasury

The most leveraged decision isn’t on this list. Put an on-chain indirection in front of every high-value identity. If a contract’s owner is a multisig, and the hot key is only an executor that the multisig can reassign with setExecutor(newAddress), then rotating the hot key is a single transaction with nothing stranded. One team’s playbook for this case is half a page. Their playbook for a key baked directly into CREATE2 initcode as initialOwner, owning every proxy, states plainly that no migration mechanism was found in any repository. The cheapest key-management control is a setter you add before deployment.

4.4 Shared vs per-environment keys

Sharing one key between dev and prod looks attractive: one fee-payer address to fund, and one less thing to provision. Avoid it.

The key’s policy must admit dev principals (humans, dev CI), so prod funds are reachable from dev.
The key is owned by one environment’s Terraform state. If dev state owns the prod fee-payer, a dev refactor can plan its replacement.
You can’t split them later without an on-chain cutover, because the address is the identity.

Rule: one key per environment per purpose. Dev keys may exempt human admins from the network Deny so that developers can sign from laptops against testnets. Prod keys exempt nobody.

4.5 Static credentials for local development

Developers will want to sign locally. Give them dev keys, on testnets, through their own MFA’d IAM identity or SSO role, listed by ARN in that dev key’s allowed_principal_arns, and exempt them from the IP Deny only on dev keys. The anti-pattern seen in the field: an IAM user per key, with its access key pasted into 1Password and shared among several developers, on a prod key. Usage then can’t be attributed, the credential can’t be rotated without coordinating everyone, and offboarding doesn’t revoke it.

5. Failure modes and field lessons

The default key policy treated as a boundary. Thirteen prod signing keys; nine had principal scoping, two were retired with a deny-all, and the most privileged one had neither a principal nor an IP restriction. Every administrator principal in the account, including an ungated CI role, could sign with it. Lesson: make allowed_principal_arns or deny_all_signing a required module input for any SIGN_VERIFY key, enforced by policy-as-code (a conftest rule on the plan JSON).
A network change broke signing. A KMS interface endpoint had private DNS enabled. Signing traffic moved off the NAT EIPs, the key’s aws:SourceIp Deny matched every call, and prod signing went down (SEV1). Lesson: any key policy with a source-IP condition must also accept the region’s KMS endpoint IDs via aws:sourceVpce. Add a CI check that flags new or modified aws_vpc_endpoint resources for kms while IP-conditioned keys exist.
Retiring a credential wedged the whole environment. Setting create_iam_user = false on three prod signers with live, hand-created access keys made the destroy 409. A push that GitHub dropped meant no apply ran right away to surface it, so the first red apply appeared about 11 hours later on an unrelated PR, and every prod apply stayed red until it was fixed. Lesson: declare credential deletion with force_destroy, in its own apply, before removing the resource.
“Remove the consumer” opened the key. Emptying an allowlist to retire a key disabled the principal Deny instead of denying everyone. Lesson: use a separate, unconditional deny-all statement. Never rely on empty-list semantics in a condition operator for a fund-moving key.
A dev/prod shared key, owned by dev. The prod Solana fee-payer was one key shared with dev, defined in dev state, with no IP Deny. Lesson: write ownership into your key inventory (which state, which account, which consumers), and review the inventory, not just the code.
A funded wallet left behind by a decommission. The Lambdas were deleted. The secret holding a private key for a funded wallet survived, readable by the PR plan role, and old state versions kept a copy even after the resource was removed from state. Lesson: a decommission checklist for anything that touches chain must include “sweep funds” before “delete secret”. Then purge old state versions, or rotate the funds away so the key is worthless.
The KMS alert that paged for nothing. A CloudTrail alert on PutKeyPolicy fired as SEV1 for an ordinary Terraform apply, because the alert carried no principal or key context and the on-call couldn’t tell routine from hostile. Lesson: KMS and Secrets Manager change alerts must group by key or secret and by principal, include an Explore/CloudTrail deep link, and suppress (or downgrade) known deploy-role principals during an apply window. Keep the alert itself, though: ScheduleKeyDeletion, DisableKey, PutKeyPolicy, and Sign by a principal outside the allowlist (which the Deny blocks, but which is still a signal) are your highest-value security detections.
The CI-exposure ticket closed with a docs change. The PR-plan secrets exposure was “fixed” by documenting it, and the real IAM change landed months later. Lesson: close security tickets only against a verifiable artefact, such as a policy diff, a plan, or a denied API call.
The once-only credential. A Connect server’s credentials file was nearly lost during cleanup. Regenerating it would have meant deleting the server and revoking every token in use. Lesson: any “issued once” credential (Connect credentials, MFA seeds, initial root passwords, some vendor API keys) goes to its durable store in the same sitting it’s issued.
Succession plans that stop at “deploy a new key”. One playbook for key succession had a real procedure for exactly one key, which sat behind a contract wrapper whose stored signer could be updated. For the two highest-blast-radius keys, the playbook recorded that no ownership-migration mechanism had been found. It also found a hidden coupling: refunds verified that a collector address persisted in the database matched the KMS-derived address, so rotating that key without migrating the stored rows would fail every refund. Lesson: for each key, list every reference to its address: on-chain owner/role slots, allowlists at counterparties, database rows, config, monitoring. Do it while the key is healthy. CCSS’s Key Compromise Policy aspect exists for exactly this [9]. NIST SP 800-57 calls these the key’s cryptoperiod and usage periods [20]. For on-chain identities, the cryptoperiod is “until we migrate”, so plan the migration.
Making a secret into a KMS key. A KMS HMAC key was provisioned for code that ran HMAC locally (section 2.5). It was caught in review, but only after the PR existed. Lesson: read the consumer first.
The audit log that doesn’t log the right thing. CloudTrail records kms:Sign calls with the caller and the key, but not the transaction. If you need “who signed what”, log the digest and decoded transaction intent in the signing service, to an append-only store, and correlate by request ID. Investigating a suspected misuse without that log means reconstructing intent from on-chain data after the fact.

6. Best-practice checklist

Day 1 (seed stage, before mainnet funds)

Every signing key is SIGN_VERIFY, non-exportable, created by a single shared Terraform module.
The module enforces prevent_destroy, deletion_window_in_days = 30, and a validated key_spec.
Every prod key carries a Principal: "*" Deny on kms:Sign/GetPublicKey for principals not in the allowlist; retired keys carry an unconditional deny-all.
Network Denies, if used, accept the KMS VPC endpoint via aws:sourceVpce.
Signing services use IRSA or Pod Identity roles. No static access keys reach prod keys.
One key per environment per purpose. No dev principal is allowed to use a prod key.
High-value identities are an on-chain multisig; hot keys are reassignable executors.
Human IAM: MFA enforced for everything except session-token calls, and self-enrollment not exempt.
GitHub Actions uses OIDC, with sub pinned to repo and branch or environment [8]. No long-lived AWS keys in CI.
No secrets in Git. Secret scanning on every PR.
CloudTrail alerts on ScheduleKeyDeletion, DisableKey, PutKeyPolicy, root sign-in, and IAM user/access-key creation.

First year (Series A)

Deploy-role guardrail Denies, including kms:Sign on every role that does not sign.
PR plan role split from the main plan role; GetSecretValue scoped to named ARNs.
No secrets in Terraform outputs; DB master credentials not in remote state; state bucket on a dedicated CMK with versioned-object lifecycle.
Ephemeral resources and write-only arguments adopted where providers support them.
Secrets split per service; a written two-phase procedure for moving references.
A key inventory listing, for each key: address(es), chain(s), owning state, allowed principals, every on-chain and off-chain reference, and its succession procedure.
One break-glass playbook per scenario, with at least the key-succession playbook drilled on a testnet.
A written, tested offboarding checklist, with a “direct grants” audit step.
Policy-as-code: CI fails if a SIGN_VERIFY key lacks a principal restriction or deny-all.
The signing service logs transaction intent (digest + decoded fields) to an append-only store.

At scale (Series B+)

Per-environment AWS accounts under Organizations with SCPs; a separate security/log-archive account [18].
IAM Identity Center for humans; no IAM users except the break-glass ones.
Egress default-deny for key-holding workloads, with a KMS VPC endpoint so signing doesn’t depend on the NAT.
Policy-enforced signing (TEE or MPC) for flows where “the app decided to sign” is not an acceptable trust assumption.
Automated rotation for every rotatable secret; expiry alerts for vendor keys.
A CCSS self-assessment against the key generation, storage, usage and compromise aspects, with evidence links [9].

7. Maturity roadmap

Stage	Signing keys	Secrets	Identity	Evidence you can show
0: Ad hoc	Raw private keys in env vars or a secret store	Mixed, partly in Git history	Shared IAM users	None
1: KMS default	KMS keys with the default root policy	Central store	IAM users + MFA	“Keys never leave the HSM”
2: Scoped KMS	Principal + network Denies; `prevent_destroy`; per-env keys; IRSA	Per-service items; CI scoped; nothing secret in outputs	MFA without self-enroll; OIDC CI; guardrail Denies incl. `kms:Sign`	Key policies, CloudTrail alerts, key inventory
3: Succession-ready	On-chain indirection for every high-value key; drilled succession playbooks	Rotation automated where possible; ephemeral TF values	Identity Center; documented break-glass, drilled	Drill logs, CCSS self-assessment
4: Structural	Per-account isolation; TEE/MPC for policy-enforced flows	Short-lived dynamic credentials	SCPs, SoD on prod applies	External audit / CCSS certification

Most teams can get from stage 1 to stage 2 in two or three focused weeks. Stage 3 is mostly smart-contract and process work, not infrastructure, which is why it gets skipped, and why its absence hurts most when it matters.

8. References

AWS KMS Developer Guide — Default key policy (account principal delegates to IAM; lockout risk). https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-default.html
AWS KMS Developer Guide — Use VPC endpoints to control access to AWS KMS resources (aws:SourceIp not effective via VPC endpoints; use aws:sourceVpce). https://docs.aws.amazon.com/kms/latest/developerguide/vpce-policy-condition.html
AWS KMS Developer Guide — Condition keys for AWS Nitro Enclaves (kms:RecipientAttestation:ImageSha384, :PCR). https://docs.aws.amazon.com/kms/latest/developerguide/conditions-nitro-enclaves.html
AWS KMS Developer Guide — Delete an AWS KMS key (7–30 day waiting period; asymmetric-key considerations). https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html
AWS Nitro Enclaves User Guide — Using cryptographic attestation with AWS KMS. https://docs.aws.amazon.com/enclaves/latest/user/kms.html
AWS Database Blog — How to use AWS KMS to securely manage Ethereum accounts, Part 1 (ECC_SECG_P256K1, SIGN_VERIFY, address derivation). https://aws.amazon.com/blogs/database/part1-use-aws-kms-to-securely-manage-ethereum-accounts/
HashiCorp Terraform — Sensitive data in state (state/plan exposure; ephemeral resources and write-only arguments). https://developer.hashicorp.com/terraform/language/state/sensitive-data
GitHub Docs — Configuring OpenID Connect in Amazon Web Services (restricting the sub claim). https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
CryptoCurrency Certification Consortium — CryptoCurrency Security Standard (CCSS). https://cryptoconsortium.org/ccss/
1Password Developer — 1Password Connect. https://www.1password.dev/connect/
External Secrets Operator — Introduction. https://external-secrets.io/latest/
AWS Well-Architected Framework, Security Pillar — SEC02-BP03 Store and use secrets securely. https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/sec_identities_secrets.html
Kubernetes Documentation — Good practices for Kubernetes Secrets. https://kubernetes.io/docs/concepts/security/secrets-good-practices/
AWS IAM User Guide — Allows MFA-authenticated IAM users to manage their own credentials (the DenyAllExceptListedIfNoMFA pattern). https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_aws_my-sec-creds-self-manage.html
AWS Whitepaper, Organizing Your AWS Environment Using Multiple Accounts — Break-glass access. https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/break-glass-access.html
Ethereum Improvement Proposals — EIP-2: Homestead Hard-fork Changes (signatures with s > secp256k1n/2 invalid). https://eips.ethereum.org/EIPS/eip-2
Amazon EKS User Guide — IAM roles for service accounts. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
AWS Prescriptive Guidance — AWS Security Reference Architecture (AWS SRA). https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/welcome.html
AWS KMS Developer Guide — AWS KMS condition keys (kms:SigningAlgorithm, kms:ViaService, kms:CallerAccount). https://docs.aws.amazon.com/kms/latest/developerguide/conditions-kms.html
NIST SP 800-57 Part 1 Rev. 5 — Recommendation for Key Management: Part 1 – General (May 2020). https://csrc.nist.gov/pubs/sp/800/57/pt1/r5/final
OWASP Cheat Sheet Series — Secrets Management Cheat Sheet. https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html