Building on AWS · guide 02 of 13

Terraform State and Module Design

7295 words · about 37 minutes · all guides

Abstract. This guide is for the engineers who own Terraform at an AWS startup with five to fifty engineers, a platform team of one to four, and probably some fund-moving infrastructure: KMS signing keys, relayers, RPC fan-out, indexers. Most such codebases start as one root module. Somewhere between seed and Series B they become a dozen roots that read each other’s state, with modules nobody versions, dev and prod files that were copied once and then drifted, and secrets sitting in state files that half the company can read. The guide covers where to draw state boundaries; how to pass data between roots without building a dependency cycle or a credential leak; how to version and test modules; and how to keep dev and prod honest without a framework you will regret. It also covers the lifecycle settings (prevent_destroy, ignore_changes, refresh behaviour) that decide whether state describes reality. The recommendations come from a mature production platform that went through all of these phases, and from the HashiCorp and AWS primary documentation. Every failure mode described below happened for real.

1. The problem space

1.1 Forces specific to startups

Small team, high fan-out. A platform team of two maintains infrastructure for twenty services across two or three environments. Copying is the cheapest move, so files get copied. Abstraction costs time up front, and the payback is hard to show in a sprint review. That is why a typical mid-stage codebase has dev and prod roots with dozens of same-named files, and why the review behind this guide found 80 variables declared in both dev and prod with identical defaults on 78 of them.

Speed over isolation, until an incident. The first shared root collects everything that doesn’t obviously belong to one environment: the CI OIDC provider, ECR, DNS, the VPN, a tooling cluster, the account’s detective controls. For two years that is the right call. Then the root holds 70+ files and 11k+ lines under one apply role, and a typo in a landing-page bucket policy plans in the same run as a change to the IAM trust policy that every deploy role depends on.

Compliance and partner pressure arrives mid-stream. SOC 2 CC8.1 (change management), exchange and custody partner questionnaires, and auditors all ask the same questions. Who can change prod? Is every change reviewed? Can a reviewed module silently change under you? Can you show that state at rest is protected? A layout designed before anyone asked will usually answer “partly”.

The crypto stakes. On most startups’ AWS bills, the most dangerous resource is the database. In a crypto company it is an aws_kms_key with key_spec = "ECC_SECG_P256K1" and key_usage = "SIGN_VERIFY". That key is an on-chain address. Its private material cannot be exported. If it is deleted, which takes a 7–30 day waiting period [18], every asset held by that address and every contract role granted to it is gone permanently. A misplaced terraform destroy, a for_each key rename without a moved block, or a module edit that reaches 25 call sites at once is not an outage. It is a loss event. The state layout has to make that class of mistake structurally hard, not merely unlikely.

1.2 What “good” looks like

Infrastructure as code promises that systems are reproducible and that changes are reviewed, versioned and repeatable [22]. A Terraform layout keeps that promise only if it has these properties:

Bounded blast radius per apply. One plan can only touch what one role can touch, and that is one tier of one environment.
An acyclic dependency graph. You can rebuild any environment from an empty account by applying roots in a written order, with no -target surgery.
No credentials in state. At minimum, none in outputs that other roots can read.
Deliberate propagation. A module change reaches prod only when someone bumps prod’s reference.
Explicit differences. Every difference between dev and prod is one greppable value with a reason next to it, not a diff between two 400-line files.
State that describes reality. You know which fields Terraform deliberately does not own, and you detect when anything else drifts.

2. Reference design

2.1 Root decomposition

Split by rate of change, blast radius and credential boundary. Do not split by AWS service. The layers below are listed in dependency order. Every arrow points one way.

The root layers in dependency order, from bootstrap to workload, split by rate of change, blast radius and credential boundary. full size ↗

Rules that make this work:

L0 is its own root and its own pipeline. The state bucket, its CMK and its access-log bucket should never plan in the same run as anything else. If that state is lost, you rebuild it with a handful of terraform import commands from a workstation. Write those commands in the root’s README now. Watch for a bootstrap race. If L0’s CI deploy role is created by L1, and one push adds both, the two applies run as unordered workflows, and the first L0 apply fails on AssumeRoleWithWebIdentity. That is harmless and happens once, but document it.
L1 is the trust root and must not be a grab-bag. IAM, OIDC, deploy roles and detective controls change a few times a quarter, need the strictest review, and should have the only apply role that can touch IAM trust policies. They do not belong next to CodeBuild projects and marketing buckets.
L3/L4 split infrastructure from workload wiring. EKS clusters and RDS instances change monthly. Helm releases and namespace policy change daily. Different cadence, different reviewers, different blast radius.
L5 roots get their own workflows. Their providers (Grafana, PagerDuty, Argo CD) often need network paths or credentials the AWS roots don’t have. One consequence you must remember: a change in an L5 directory is applied by a different, later workflow run. “Merged” does not mean “applied” until you have found that run.

At seed stage this collapses to three roots: bootstrap, prod, dev. By Series A you want roughly the full picture. Section 3.1 says when to cut.

2.2 Backend

One bucket per account (or per environment, once you have accounts), one key per root, S3-native locking, a dedicated CMK:

terraform {
  required_version = ">= 1.11"
  backend "s3" {
    bucket       = "acme-tfstate-prod"           # per-account bucket
    key          = "prod/foundation.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:eu-west-1:111111111111:key/…"   # dedicated state CMK
    use_lockfile = true                           # S3-native lock; DynamoDB locking is deprecated [4]
  }
}

Bucket hardening, which all lives in L0:

resource "aws_s3_bucket" "state" {
  bucket = "acme-tfstate-prod"
  lifecycle { prevent_destroy = true }
}
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}
resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
# Bucket policy: deny aws:SecureTransport=false; deny PutObject without the CMK;
# grant read only to the plan/apply roles of the roots that need it.
resource "aws_kms_key" "state" {
  description         = "Terraform state encryption"
  enable_key_rotation = true
  lifecycle { prevent_destroy = true }   # losing this key = losing every state file
}

The kms_key_id matters for more than encryption-at-rest checkboxes. With SSE-S3, anyone with s3:GetObject can read state. With a CMK, reading state also needs kms:Decrypt on the key, so the key policy becomes a second, independently reviewable access list [4][13]. AWS Prescriptive Guidance also recommends CloudTrail data events on the state bucket and alerts on state access from workstations rather than CI [13]. Most teams skip that until an auditor asks.

2.3 Cross-root data: a one-way contract over SSM

Do not use terraform_remote_state as the default coupling mechanism. Publish a small, explicit contract:

# Producer: prod/foundation (L3)
resource "aws_ssm_parameter" "contract" {
  for_each = {
    "vpc_id"               = module.network.vpc_id
    "private_subnet_ids"   = join(",", module.network.private_subnet_ids)
    "eks_cluster_name"     = module.eks.cluster_name
    "eks_oidc_provider"    = module.eks.oidc_provider_arn
    "db_endpoint"          = module.db.proxy_endpoint      # an endpoint, never a URI with credentials
    "db_app_secret_arn"    = module.db.app_user_secret_arn # a pointer to a secret, not the secret
  }
  name  = "/platform/prod/foundation/${each.key}"
  type  = "String"
  value = each.value
}

# Consumer: prod/apps (L4)
data "aws_ssm_parameter" "foundation" {
  for_each = toset(["vpc_id", "eks_cluster_name", "db_endpoint", "db_app_secret_arn"])
  name     = "/platform/prod/foundation/${each.key}"
}
locals {
  foundation = { for k, p in data.aws_ssm_parameter.foundation : k => p.value }
}

This beats terraform_remote_state for three reasons:

Least privilege. HashiCorp’s own documentation warns that any principal who can read a root’s outputs through terraform_remote_state “will also always have access to the full state snapshot data” [1]. That includes every resource attribute, generated passwords among them. With SSM, a consumer role gets ssm:GetParameter on /platform/prod/foundation/* and nothing else [16].
The contract is explicit and greppable. The parameter path is the interface. Renaming an output in the producer doesn’t silently break consumers at plan time, because the parameter keeps existing until someone removes it deliberately.
It breaks cycles. A “reverse edge” is an upstream root that needs a value from a downstream one, for example the shared root needing an Argo CD role ARN created in prod. With remote state that edge creates a cycle. With SSM, the downstream root writes the parameter, and the upstream root reads it with a data source that can fall back to a default on first build. Or better, the upstream root gets the value from a naming convention and never reads it at all.

Secrets never go in this contract. Pass an ARN, and let the workload, or an ephemeral resource at apply time, resolve it.

2.4 Secrets and state

Mark it on the wall: sensitive = true does not keep a value out of state. It hides the value from CLI output. Terraform still writes it to state and plan files, and “anyone who can access those files can access your sensitive values” [5]. The tools that actually keep secrets out of state:

Need	Mechanism	Since
Read a secret only during apply (e.g. to configure a provider)	`ephemeral "aws_secretsmanager_secret_version"`	Terraform 1.10 [5]
Set a password on a resource without storing it	write-only arguments (e.g. `password_wo` + `password_wo_version`)	Terraform 1.11 [5]
Let AWS own the master credential entirely	`manage_master_user_password = true` on RDS	provider feature
Encrypt state client-side regardless of bucket ACLs	OpenTofu state encryption with an `aws_kms` key provider	OpenTofu [8]

Pattern for a database:

resource "aws_db_instance" "main" {
  # …
  manage_master_user_password = true            # master secret lives only in Secrets Manager
  iam_database_authentication_enabled = true
}
# Per-application roles, created by a migration job or a postgres provider
# using ephemeral credentials. Never output a connection URI.
output "db_proxy_endpoint" { value = aws_db_proxy.main.endpoint }

The anti-pattern this replaces is common enough to name. A root outputs postgresql://<master_user>:<master_password>@replica:5432/db marked sensitive, “for the analytics job”. From then on, every root and CI role that reads that state has the master password for the primary. Another variant: a root creates an IAM user with an access key “for a vendor integration” and outputs the secret key. If that key is in state, it is in every reader’s hands. Grep for master_password, secret_access_key, urlencode( and format("postgres across your outputs. Every hit is a finding.

2.5 Modules: sourcing and versioning

Use local paths for intra-root factoring: helpers used once or twice inside a single root. Use pinned remote sources for shared modules called from several roots. AWS Prescriptive Guidance draws the same line [6].

module "signer_key" {
  source = "git::https://github.com/acme/infra.git//modules/signing-key?ref=3f2c9a1e…"  # commit SHA
  # …
}

Three rules:

ref= is a commit SHA, never a tag. Git tags are mutable. Anyone who can push tags can change what an environment applies with no Terraform diff and no reviewed ref bump [7][6]. Keep human-friendly tags (modules-2026-q3, signing-key-v1.4.0) as labels for discussion. The source line trusts only the SHA. AWS’s guidance shows SHA pinning for exactly this supply-chain reason [6].
Bump per environment. Merge the module change, bump dev’s ref, watch it, then bump prod’s. The bump PR is the reviewable change-management record. Without it, a local-path edit reaches every caller on its next plan.
Modules declare required_providers with both bounds (>= 6.0, < 7.0) and never configure providers. A module that uses the kubernetes, tls and null providers but declares only aws works by accident until a root’s lockfile changes [6].

Your CI then needs to fetch a private repo during terraform init. A url.<authenticated>.insteadOf https://github.com/ rewrite using the job’s own short-lived token is enough, so no new secret is required. Note that Docker-based Terraform actions have their own $HOME, so write the git config to a path inside the workspace and point GIT_CONFIG_GLOBAL at it.

2.6 Environment parity

Write one composite module per environment tier, and give each environment a thin root that calls it with an explicit, commented variable set:

# environments/prod/foundation/main.tf
module "foundation" {
  source = "git::https://github.com/acme/infra.git//stacks/env-foundation?ref=<sha>"

  environment          = "prod"
  vpc_cidr             = "10.30.0.0/16"   # MUST differ from dev (10.20/16): overlap makes peering impossible
  single_nat_gateway   = false            # prod: one NAT per AZ; signer IP allowlists list all three EIPs
  db_backup_retention  = 30               # dev: 1
  db_deletion_protection = true
  psa_enforce          = true             # dev: false until the preview stack is compliant
}

Put real per-environment differences in a prod.tfvars/dev.tfvars or in the thin root’s module call. Never leave them as literals buried in resource files. Composite modules must not have “safe for dev” defaults: deletion_protection = false and skip_final_snapshot = true as module defaults mean the one caller that forgets to override them is the one that loses data. Make security-relevant inputs required (no default) or default them to the safe value.

2.7 The pipeline around it

The pipeline around the roots: format and validate, plan per changed root, policy check, review, apply. full size ↗

The CI/CD guide (03) covers roles and gates. Three structural points belong here. PR plans and drift plans differ by exactly one setting, refresh, and Section 4.6 explains why that is load-bearing. Policy should run on the plan, not only on the HCL. And something must check that every merge to main actually produced an apply run.

3. Decisions and tradeoffs

3.1 How many roots, and where to cut

Option	Upsides	Downsides	Fits
One root per environment	Trivial mental model; no cross-root data	Every plan touches everything; slow plans (>5 min at ~1,500 resources); one role for IAM + apps	Seed, < 300 resources
Base / apps split per env (+ shared)	Separates monthly infra from daily workload wiring; smaller plans	Cross-root data appears; `shared` becomes a dumping ground	Series A
Layered roots (Section 2.1) per account	Blast radius matches credential boundary; reviewers match layers	More pipelines; ordering must be documented and enforced	Series A+, or any company with fund-moving keys
Fine-grained “stacks” via Terragrunt	Hundreds of small units, generated backends, `dependency` blocks	Another tool and DSL; dependency resolution reads outputs, so the remote-state concerns carry over	Many accounts × regions × services

Signals that it’s time to cut a root:

Its plan takes more than ~5 minutes.
More than one team reviews changes to it.
It holds both IAM trust policies and anything changed weekly.
Its apply role is exempt from a guardrail Deny “because it needs to manage the thing the guardrail protects”. That exemption is the signal: the trust root wants its own root and its own role.
More than ~50 files.

How to cut without downtime: use moved blocks inside a root [9]. To move resources across roots, use terraform state rm in the old root and an import block in the new one, landed in a single change window, with both plans reviewed together. Test it on dev first. A removed { lifecycle { destroy = false } } block in the old root is the declarative version of state rm.

3.2 Coupling between roots

Mechanism	Least privilege	Cycle-safe	Explicit contract	Cost
`terraform_remote_state`	No: reader gets the whole state [1]	No	Implicit (all outputs)	Zero setup
SSM Parameter Store	Yes: per-path IAM [16]	Yes, with writer-owns-parameter	Yes	Standard tier free (10k params, 4 KB) [16]
Provider data source lookup (`data "aws_vpc"` by tag)	Yes	Yes	Weak (tags are conventions)	Tag discipline
Naming convention (computed ARN)	Yes	Yes, no read at all	Implicit	Fails if names change
`tfe_outputs` (HCP Terraform)	Outputs only [1]	No	Yes	Requires HCP

Recommendation: use SSM for anything another root consumes, and data-source lookups for AWS-native discoverable resources. Use naming conventions for IAM role ARNs that must exist before their creator does, for example a cross-account trust. Keep terraform_remote_state only for low-sensitivity roots, and only in the forward direction.

Cycles are the thing to audit. Draw the graph: for each root, list every state key it reads. In the platform this guide draws on, the shared root read dev, prod and prod-apps state, while dev and prod both read shared state. That is a direct cycle. The independent review’s verdict: “a from-scratch rebuild needs hand-staged -target applies”, and that sequence was written down nowhere. You don’t find out until disaster-recovery day. The fix is mechanical. For each edge that points “up” the layer diagram, have the lower root write an SSM parameter, then replace the terraform_remote_state read in the upper root with a data "aws_ssm_parameter" read.

3.3 Module versioning

Option	Staged rollout	Immutability	Friction	Tooling caveats
Local path (`../../modules/x`)	None: every caller changes on next plan	n/a	Lowest	Scanners resolve callers natively
Git source, tag ref	Yes	No: tags can move	Low	Scanner may lose caller attribution
Git source, SHA ref	Yes	Yes	Medium (SHA bumps)	Same as above
Private registry, semver `version`	Yes, with constraints	Registry-dependent	Medium; needs registry	Best UX for many repos
Separate repo per module	Yes	With SHA pin	Highest; cross-repo PRs	Clean ownership

Recommendation for a single-repo startup: monorepo, git sources with SHA refs, converted module by module, starting with the modules whose fan-out × criticality is highest. In the reference platform those were the Argo CD application module (39 callers) and the signing-key module (25 callers, fund-moving). Leave single-caller helpers on local paths.

Revisit when: you have more than one infra repo consuming the same modules (move to a registry), or SHA bumps across 30 call sites become toil (generate them with a script, or adopt Renovate’s Terraform manager).

A hidden cost to plan for. Static scanners such as checkov resolve a local-path module back to its callers, but for a git:: source they may report findings as a standalone file with no caller attribution. If your scanner runs as a ratchet against a baseline keyed on (file, caller, resource, check), pinning a module silently changes every key. Either regenerate the baseline in the same PR as each conversion, or reconstruct caller attribution yourself. Don’t attribute a finding to all callers while their refs differ: during a staged rollout dev and prod run different code, and pre-baselining prod against dev’s code hides a regression when prod bumps.

3.4 Module testing

Tool	What it tests	Cost	Use for
`terraform validate` + `tflint`	Syntax, provider schema, lint rules	Seconds	Every module, every PR. Day 1.
`terraform test`, `command = plan` with `mock_provider`	Variable validation, conditional logic, computed names/policies	Seconds, no cloud	Modules with logic: key policies, `for_each` maps, conditionals [3]
`terraform test`, `command = apply`	Real resources in a sandbox account	Minutes, real cost	Modules whose correctness is only observable live (IAM evaluation, KMS policies)
Terratest (Go)	Arbitrary post-apply assertions: call the API, sign a payload, curl the endpoint	Highest	Fund-moving or security-critical modules [10]

A useful first test for a signing-key module, which needs no cloud, asserts on the generated key policy:

# modules/signing-key/tests/policy.tftest.hcl
mock_provider "aws" {}

variables {
  name                 = "relayer-test"
  signer_role_arns     = ["arn:aws:iam::111111111111:role/signer-svc"]
  allowed_source_vpces = ["vpce-0abc"]
}

run "key_is_secp256k1_sign_only" {
  command = plan
  assert {
    condition     = aws_kms_key.this.key_spec == "ECC_SECG_P256K1" && aws_kms_key.this.key_usage == "SIGN_VERIFY"
    error_message = "signing keys must be secp256k1 SIGN_VERIFY"
  }
}

run "policy_denies_outside_vpce" {
  command = plan
  assert {
    condition     = strcontains(aws_kms_key.this.policy, "aws:SourceVpce")
    error_message = "key policy must deny kms:Sign from outside the allowed VPC endpoints"
  }
}

run "rejects_wildcard_principal" {
  command = plan
  variables { signer_role_arns = ["*"] }
  expect_failures = [var.signer_role_arns]
}

Mock providers arrived in Terraform 1.7 [3]. Before that, plan-mode tests needed credentials. Many teams have zero .tftest.hcl files and rely only on static scanners. That is the most common gap at Series A, and the cheapest to close: one test file for each of your three most-called modules.

3.5 Environment parity

Option	How differences are expressed	Strength	Failure mode
Hand-duplicated roots	Literals inside copied files	Zero abstraction; easy first step	Silent divergence; 80 duplicated variables; “fixed in prod, not dev”
One root + workspaces	`terraform.workspace` conditionals	Single codebase	Same backend and credentials for all envs. HashiCorp says workspaces are “not appropriate for … deployments requiring separate credentials and access controls” [2]
One root + per-env `.tfvars`	Values files	Clear diffs; AWS’s recommended layout [6]	Still one backend config per env to manage; easy to apply the wrong tfvars
Thin per-env roots + composite modules	Module inputs, one per difference	Separate state/credentials and shared logic	Up-front extraction cost; composite modules can grow god-object inputs
Terragrunt	`inputs` in a hierarchy of HCL files	DRY backends and inputs at scale [11]	Extra tool; team must learn it; debugging is two layers deep

Recommendation: use thin per-env roots with composite modules, plus a short env.auto.tfvars per root for scalar differences. Reach for Terragrunt only when the number of (account × region × stack) combinations runs past ~20. The SRE Workbook’s advice on configuration design applies directly: minimise the questions each caller must answer, give safe defaults, and let power users override explicitly [21].

How to extract without causing an incident. Field lessons from two extraction passes on a real platform:

Raw diff size is a poor proxy for extraction cost. The two files that turned out riskiest to extract had among the smallest diffs (4 and 29 lines). The network module call was trivial, but 17 other files in the same root read module.network.* directly. Wrapping it meant rewriting every reference in both environments. Before extracting, run grep -rl for every resource address the file declares.
Classify each duplicated pair as near-identical or structurally diverged. Near-identical pairs (metrics-server, log shipper, kube-state-metrics, flow-log bucket: diffs of 0–19 lines) extract mechanically. Structurally diverged pairs, where the two copies use different mechanisms for the same job, need a design decision first. In the reference platform, dev resolved DB credentials through a Secrets Manager data source with a fallback map, while prod read them from another root’s remote-state outputs. Folding “which pattern wins” into a mechanical refactor PR is how a refactor becomes an incident.
Some divergence is correct. Prod sets Performance Insights, IAM DB auth, 30-day backups. Dev uses one NAT gateway, so its signer IP allowlist has one IP, while prod’s has one per AZ. Those become explicit variables, never “equalised”.
A variable can beat a module. A namespace whose Pod Security admission level differed between envs was read by 30 resources across the two roots. Wrapping the namespace in a module meant 30 rewrites. Replacing two hand-written label blocks with one var.psa_enforce per environment, documented in that environment’s variables.tf, captured the whole benefit at none of the cost.
Keep a parity inventory. List one-sided files with a reason each, and duplicated files with their extraction status. Mark it “not CI-enforced; re-derive before trusting”. It is the only answer to “is this difference deliberate?”

Signals to revisit: a bug fixed in one environment and found unfixed in the other; a new environment, where the cost of a third copy is the trigger to extract; an auditor asking you to prove dev and prod differ only in documented ways.

3.6 Policy-as-code: on HCL or on the plan?

Input	Sees	Misses
HCL (conftest `hcl2` parser [12], checkov)	Structure: “every `kubernetes_namespace` has a default-deny NetworkPolicy in the same file”, “every `provider "aws"` has `default_tags`”	Computed values, module-expanded resources, actions (create/replace/destroy)
Plan JSON (`terraform show -json`, then OPA [14])	Actions and final values: “no `delete` on `aws_kms_key`”, “no replace of `aws_db_instance`”, “no IAM policy with `Action: *` on a signer role”	Anything the plan doesn’t show, such as timing and ordering across roots

Start with HCL policies for cross-cutting structural rules that per-resource scanners can’t express. A provider block without default_tags silently untags every resource created through it, and no per-resource scanner can say “the thing that sets every resource’s tags is missing”. Add plan-JSON policies for destructive actions on crown jewels. They are the guardrail that catches a for_each rename:

package terraform.plan

crown_jewels := {"aws_kms_key", "aws_db_instance", "aws_route53_zone", "aws_s3_bucket"}

deny contains msg if {
  rc := input.resource_changes[_]
  crown_jewels[rc.type]
  "delete" in rc.change.actions
  not exempt(rc.address)
  msg := sprintf("%s would be destroyed (%v); add an approved exemption or a moved block", [rc.address, rc.change.actions])
}

exempt(addr) if data.exemptions[addr]

Introduce every new policy as a ratchet. Commit a baseline of existing violations, fail only on new ones, and shrink the baseline deliberately. A policy that fails on day one with 140 findings gets disabled by day three.

3.7 Accounts versus roots

Roots bound what a plan touches. Accounts bound what a credential can touch. In a single AWS account, every deploy role is effectively one policy edit away from every other environment. The usual mitigation is an inline Deny on each role for irreversible or cross-environment actions (kms:ScheduleKeyDeletion, iam:CreateAccessKey, trust-policy edits, cloudtrail:StopLogging). It works, but it is a patch. AWS’s position is unambiguous: “Account-level separation is strongly recommended” [15][17]. The independent review of the reference platform traced most of its critical and high security findings to the single account. Guide 01 covers the move. For Terraform, the consequence is that a well-layered root structure makes the account migration a matter of re-pointing backends and providers. A grab-bag root makes it a rewrite.

4. Failure modes and field lessons

Each of these happened on a real production platform. Names and details are generalised.

4.1 The grab-bag `shared` root

What happened. A shared root grew to 74 files and 11.5k lines. It held human IAM, the CI OIDC provider and deploy-role guardrails, GuardDuty/Config/Access Analyzer, 26 near-identical CodeBuild files, ECR, landing-page buckets, DNS zones, the VPN and a tooling EKS cluster. Its deploy role was also the one exempt from the “don’t tamper with deploy-role trust” Deny, because it owned those roles.

Why it matters. Any change to a marketing site’s bucket ran under a role that could rewrite the trust policy of every deploy role in the account. Every apply of that root put the account’s trust root at risk.

Fix. Split into account-baseline (IAM/OIDC/detective controls), ci (ECR + build runners) and edge (DNS/CDN/sites), and keep network/VPN/tooling-cluster in shared. Give each its own role, so that only account-baseline is exempt from the trust Deny. The 26 CodeBuild files collapse into one for_each map. A related misplacement: an account-wide CloudTrail trail living in prod state. Account-wide resources belong in the account baseline.

4.2 Cycles discovered on rebuild day

What happened. Shared read prod and dev state, and dev and prod read shared state. Each edge made sense when it was added. Nobody drew the graph until a DR review asked “rebuild prod from an empty account: in what order?” The answer was a hand-sequenced series of -target applies that existed in nobody’s head in full.

Fix. Section 3.2: invert reverse edges into SSM parameters, then write the rebuild order down and drill it on dev. Generate the graph in CI (grep terraform_remote_state keys per root, fail on a cycle) so the next reverse edge is caught at PR time.

4.3 Credentials in outputs

What happened. A read-replica connection URI built with format() from the module’s master username and urlencode(master_password) was exported as an output for an analytics consumer. Separately, a root created a static IAM access key and output the secret. Both outputs were sensitive = true, which protected them from nobody who could read state [1][5].

Fix. Per-application DB roles with IAM auth or Secrets Manager-managed passwords. Outputs carry endpoints and secret ARNs, never credentials. Replace static IAM keys with role assumption or Roles Anywhere. Then rotate everything that was ever in state, because state history is versioned and old versions keep the old values.

4.4 One module edit, every environment, immediately

What happened. 125 of 126 module call sites used local paths. A change to the signing-key module, which had 25 callers, all fund-moving, would reach dev and prod on their next plans with no way to stage it. Nobody had chosen that blast radius. It is simply what a local path means.

Fix. SHA pinning, converted module by module (Section 3.3). Bump dev first, then prod. Where a module is still on a local path, at least make the plan-JSON destroy policy (Section 3.6) mandatory for the roots that call it.

4.5 Insecure module defaults

What happened. A database module defaulted deletion_protection = false and skip_final_snapshot = true, which suited dev. Prod overrode both. A new prod database created by copying a dev call site would have inherited the dev values.

Fix. Security-relevant module inputs default to the safe value or have no default. Dev opts out explicitly, in a line a reviewer will see.

4.6 Refresh-skipped plans and stale state

PR plans commonly run with refresh = false: the question is “what does this diff change”, and skipping refresh keeps the gate fast. That is a legitimate tradeoff with three consequences teams discover the hard way.

Drift detection must refresh. With refresh off, Terraform compares config against stored state and cannot see a resource changed outside Terraform. That is exactly what drift is. A refresh = false scheduled plan reports “no changes” forever, which is worse than no report. Expect the first refreshing runs to surface a backlog. Triage each finding into an allowlist with a reason, or fix it.

After a partially failed apply, state lies. In one case, an apply kept failing on a CloudFront Function update with 412 PreconditionFailed. An earlier partial apply had pushed a real update, but the run failed before state recorded the new ETag, so every retry sent a stale If-Match. In the same root, state showed an ACM certificate PENDING_VALIDATION while AWS reported it ISSUED, and two deposed instances had piled up. The right move was not a code fix. It was terraform apply -refresh-only [19], which makes zero infrastructure changes and re-syncs state. Rule of thumb: when an apply fails the same way twice on the same resource, compare live API state to state before writing code.

Secrets passed as variables can’t be dropped from PR plans. If a root sets a Kubernetes Secret or Helm value from TF_VAR_x, and PR plans don’t refresh, passing an empty value to a job that doesn’t need the secret plans a change to that secret. Scoping secrets to the jobs that need them takes deliberate wiring per root, not a blanket removal.

Some roots can’t refresh at all from CI. A root whose provider points at a VPN-only endpoint (an internal Grafana, for example) produces a permanently red drift job unless CI has a network path. Record such a root as “no drift coverage” rather than leaving a red job everyone learns to ignore.

4.7 `ignore_changes`: when state stops describing reality

ignore_changes attributes are considered on create and ignored on update [20]. That is the point, and also the trap.

The config change that silently does nothing. An EKS managed node group with ignore_changes = [scaling_config[0].desired_size], because the autoscaler owns it. An engineer raised min_size from 0 to 1 and also bumped desired_size in config, expecting both to apply. Terraform omitted desired_size from the API call. EKS enforces min ≤ desired ≤ max and rejected min=1 against a live desired=0. The mirror image: lowering max_size to the config’s min_size while live desired was 2 under real load. The fix: before changing min/max on such a group, read live scalingConfig from the API. Where the change must push a new desired size, add an explicit reconcile step (a terraform_data with a local-exec calling update-nodegroup-config) rather than trusting the variable.
Ignored fields drift. A VPN instance with ignore_changes = [user_data], set so that adding a peer doesn’t replace the instance, means the peer list in config and the peer list on the box diverge with no signal. If you must ignore a field, add a check that compares the two: a check block, a drift script, or a config-hash tag.
State is not an inventory. Any field under ignore_changes must be read from the live API when it matters. Write that next to the lifecycle block.

4.8 `prevent_destroy` is weaker than people think

prevent_destroy blocks a plan that would destroy the resource. It does not prevent destruction if you remove the resource block from configuration [20]. Delete the block, and the next plan destroys the resource with no complaint. A guardrail-on-the-config cannot protect against deleting the config.

What actually protects crown jewels:

prevent_destroy on every KMS key (signing and encryption), the state bucket and its CMK, apex Route53 zones and DNSSEC KSKs, and prod databases. The review found apex zones, DNSSEC keys and the state CMK without it, while far less critical resources had it.
A plan-JSON policy that fails on any delete action against those types (Section 3.6). This does catch “block removed”, because the plan still shows the delete.
An IAM Deny on kms:ScheduleKeyDeletion / DisableKey / PutKeyPolicy for every deploy role that has no business doing so (guide 07).
The provider-side flags: RDS deletion_protection, and not setting force_delete = true on ECR repositories. With force_delete on a for_each of repos, renaming a map key destroys the repository and the images prod is running.

4.9 Decommissioned environments are loaded guns

What happened. A staging environment was torn down, except two KMS signing keys that couldn’t be destroyed because they controlled on-chain roles. The directory survived with just the backend, provider and key definitions, and its state held only those keys. Anyone “restoring” config into that directory and applying would recreate the entire environment, databases included, on top of dev’s shared instances.

Fix. Remove the directory from every CI matrix so nothing can plan or apply it. Leave a README explaining what remains and why. Record the keys’ addresses and residual balances in the DR inventory. Treat “sweep funds, then schedule deletion” as an explicit, ticketed end-of-life step, not something a future terraform destroy does.

4.10 Ordering across roots and GitOps

Merges land atomically in git, but applies don’t. Three ordering traps recur.

Config and its consumer in one merge. An alert rule and the scrape job it depends on lived in different roots with different workflows. The rule root applied faster, so the rule went live nine minutes before its metric existed and paged on NoData. Rule: when root A consumes something root B creates, merge B first, verify it, then merge A.
Moving values between files that GitOps reads. Argo CD reads values files from git HEAD the instant you merge, but the Application’s valueFiles list is Terraform-managed and changes only when that root applies, minutes later. Moving a key from values-prod.yaml into a new values-common.yaml in the same merge that adds the file to the list left the key in no file the Application was reading. A required env var vanished, and the service crash-looped for six minutes. Neither the terraform plan nor a rendered-manifest diff showed it, because both describe the end state, not the transition. Two merges: add the file and the list entry; wait for the apply; then delete the duplicates.
Variables read by config applied elsewhere. A service config applied by Terraform read ${VAR}s whose values were synced by GitOps, and the service refused to boot on an empty expansion. Adding a variable: the env var lands first, the config that reads it second. Removing one: the reverse.

The general lesson: for any change spanning two pipelines, a clean plan is not evidence. Write the transition state down and sequence the merges.

4.11 Applies that silently didn’t happen

No workflow run at all. On one occasion the CI platform created no workflow run for a merge commit. The apply workflow diffed only its own commit, so the next merge didn’t pick the change up either. A key rotation’s GitOps half applied while its Terraform half never did, and a dependent indexer crash-looped on 401s. Fix: a scheduled “apply coverage” check confirming that every merge to main touching a root has a successful apply for that root.
A cancelled matrix leg. An unrelated prod failure cancelled the in-flight prod-apps apply in a max-parallel: 1 matrix. Default-deny NetworkPolicies, the enforcement half, stayed missing for weeks while the allow rules were live. Fix: a post-apply verification step that asserts the presence of security-critical objects, and a gate that refuses later applies to an environment while its last apply from another commit is not success.

4.12 Small things that cost hours

Default CIDRs in the network module. Neither env set vpc_cidr, so both used the module default 10.20.0.0/16. Peering to prod became impossible, and prod access had to go over PrivateLink. Make the CIDR a required input.
A plan role with an explicit secret allow-list. Any new aws_secretsmanager_secret_version data source a plan reads must be added to that list in the same PR. Otherwise PR plans, or the daily drift plan on refresh, fail with AccessDenied. It’s a good control, but document it next to the list.
A per-environment override script that rewrote a Terraform variable’s default for one environment. Engineers trusted variables.tf and were wrong. If a wrapper script can override variables, grep it before trusting a default.
Dead modules and dead resources. A module with zero callers, an unreferenced security group open to 0.0.0.0/0, a count = 0 release, and a client_vpn_enabled = true flag with no Client VPN that still opened an ingress rule. Each of these is noise in every review. Delete them.
“Temporary” is permanent. Thirteen redundant IAM policies marked “removed in a follow-up” were still live months later. Track TEMPORARY comments as a list with owners and dates.

5. Best-practice checklist

Day 1 (seed, before the first production key)

Remote S3 backend, use_lockfile = true, versioning, public-access block, TLS-only bucket policy, dedicated CMK with prevent_destroy [4][13].
State bootstrap in its own root, with import commands for a lost-state rebuild in its README.
Separate state keys and separate plan/apply roles per environment. No workspaces for env separation [2][13].
prevent_destroy on every KMS key, the state bucket and CMK, apex DNS zones, prod databases.
No credentials in outputs. RDS manage_master_user_password = true [5].
required_providers with bounds in every module. Lockfiles committed.
fmt/validate/tflint on every PR [6].
Network CIDR is a required input, unique per environment.
Security-relevant module inputs default to safe values.

First year (Series A)

Layered roots: bootstrap → account baseline → shared (ci/edge/net) → env foundation → env apps → gitops/observability.
Cross-root data over SSM parameters. Remote-state graph is acyclic and checked in CI.
Rebuild order written down and drilled on dev.
Shared modules pinned by commit SHA, bumped dev-then-prod [7][6].
terraform test with mock providers on the three most-called modules; Terratest or apply-mode tests on the signing-key module [3][10].
Plan-JSON policy denying destroy on crown-jewel types; structural HCL policies; both ratcheted [12][14].
Daily drift plan with refresh = true, triaged allowlist, and a list of roots with no coverage [19].
Apply-coverage check (every merge has an apply) and post-apply verification of security-critical objects.
Dev/prod parity inventory; near-identical pairs extracted; diverged pairs have a recorded design decision.
Ephemeral resources / write-only arguments for any secret Terraform must handle [5].
Every ignore_changes carries a comment saying who owns the field and how to read its live value.

At scale (Series B+)

Per-environment AWS accounts under Organizations with SCPs; one state bucket and CMK per account [15][17].
Private module registry, or Renovate-driven SHA bumps.
Composite “stack” modules per tier; thin per-env roots; consider Terragrunt only past ~20 stack instances [11].
CloudTrail data events on state buckets; alert on non-CI state access [13].
Client-side state encryption (OpenTofu) if your threat model includes bucket-level compromise [8].
Scoped least-privilege apply roles derived from real plan output, replacing admin-plus-Deny.

6. Maturity roadmap

Stage	State and roots	Modules	Parity	Safety nets
0: Seed	One bucket, CMK, per-env roots (`dev`, `prod`), bootstrap root	Local paths; a few modules	Copy-paste, but CIDRs and security defaults are variables	`prevent_destroy` on keys/state/DB; no secrets in outputs
1: Product-market fit	Base/apps split; `shared` root appears; remote state in use	First shared modules (DB, signing key, EKS)	Parity inventory started	PR plans in CI; checkov ratchet; approval on prod applies
2: Series A	Layered roots; `shared` split into baseline/ci/edge/net; SSM contracts; acyclic graph	SHA-pinned shared modules; `terraform test` on top modules	Composite modules for env foundation; tfvars for scalars	Plan-JSON destroy policy; daily refreshing drift plan; apply-coverage check
3: Scale	Account per env; per-account state; workload roots per team	Registry or Renovate; apply-mode tests in a sandbox account	Stack modules; optional Terragrunt	Least-privilege apply roles; state-access alerting; drilled from-scratch rebuild

Order of operations when retrofitting an existing codebase. The fastest risk reduction per engineer-day comes first:

Remove credentials from outputs and rotate them.
Add prevent_destroy and the plan-JSON destroy policy to crown jewels.
Turn on refreshing drift detection.
Break remote-state cycles with SSM.
Split the grab-bag root.
Pin the highest-fan-out modules.
Extract near-identical dev/prod pairs.
Move to multiple accounts.

Steps 1–3 take days. Steps 4–7 take weeks each and can run in parallel. Step 8 is a quarter, and is much cheaper if steps 4–5 are done.

7. References

HashiCorp. The terraform_remote_state data source, including the warning on full-snapshot access and alternatives (SSM, tfe_outputs). https://developer.hashicorp.com/terraform/language/state/remote-state-data
HashiCorp. Workspaces: “not appropriate for system decomposition or deployments requiring separate credentials and access controls”. https://developer.hashicorp.com/terraform/language/state/workspaces
HashiCorp. Tests (.tftest.hcl, command = plan|apply, mock providers since v1.7). https://developer.hashicorp.com/terraform/language/tests
HashiCorp. Backend type: s3 (use_lockfile, DynamoDB locking deprecation, kms_key_id). https://developer.hashicorp.com/terraform/language/backend/s3
HashiCorp. Manage sensitive data: sensitive values are stored in state; ephemeral values (1.10) and write-only arguments (1.11) are not. https://developer.hashicorp.com/terraform/language/manage-sensitive-data
AWS Prescriptive Guidance. Best practices for using the Terraform AWS Provider: code base structure and organization (module sources, SHA pinning, required_providers, tfvars layout, TFLint). https://docs.aws.amazon.com/prescriptive-guidance/latest/terraform-aws-provider-best-practices/structure.html
HashiCorp. Module sources (git ref by tag or SHA, registry version). https://developer.hashicorp.com/terraform/language/modules/sources
OpenTofu. State and plan encryption (AES-GCM; aws_kms key provider). https://opentofu.org/docs/language/state/encryption/
HashiCorp. Refactor modules (moved blocks, Terraform ≥ 1.1). https://developer.hashicorp.com/terraform/language/modules/develop/refactoring
Gruntwork. Terratest documentation. https://terratest.gruntwork.io/docs/
Gruntwork. Terragrunt quick start (units, stacks, dependency blocks). https://docs.terragrunt.com/docs/getting-started/quick-start
Open Policy Agent. Conftest (HCL2 parsing, Rego policies for configuration files). https://www.conftest.dev/
AWS Prescriptive Guidance. Backend best practices (S3 native locking, versioning, CloudTrail on state, separate backends per environment, restricting prod state). https://docs.aws.amazon.com/prescriptive-guidance/latest/terraform-aws-provider-best-practices/backend.html
Open Policy Agent. Terraform (evaluating terraform show -json plan output). https://www.openpolicyagent.org/docs/latest/terraform/
AWS Well-Architected Framework, Security Pillar. SEC01-BP01 Separate workloads using accounts. https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/sec_securely_operate_multi_accounts.html
AWS. AWS Systems Manager Parameter Store (types, tiers, IAM path scoping, Secrets Manager for credentials). https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html
AWS Whitepaper. Organizing Your AWS Environment Using Multiple Accounts. https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html
AWS. Delete an AWS KMS key (7–30 day waiting period; deletion is irreversible). https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html
HashiCorp. Use refresh-only mode to sync Terraform state. https://developer.hashicorp.com/terraform/tutorials/state/refresh
HashiCorp. The lifecycle meta-argument (prevent_destroy does not apply when the block is removed; ignore_changes semantics). https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle
Google. The Site Reliability Workbook, Ch. 14: Configuration Design and Best Practices. https://sre.google/workbook/configuration-design/
Martin Fowler. InfrastructureAsCode (bliki). https://martinfowler.com/bliki/InfrastructureAsCode.html