Building on AWS · guide 01 of 13

AWS Account Strategy and Network Topology for Startups

7927 words · about 40 minutes · all guides

Abstract. This guide is for senior DevOps, SRE and platform engineers at startups on AWS, especially crypto and web3 infrastructure companies. These teams hold signing keys, run relayers and indexers, settle on-chain, and usually have one or two people doing all of the infrastructure work. It covers two decisions that are cheap to get right in month one and very expensive to change in year two. The first is how many AWS accounts you run, and how they are governed. The second is how your VPCs are addressed and connected, including how humans reach private resources.

You will get an opinionated reference design: an AWS Organizations layout, an address plan, a hub-and-spoke “shared services” VPC, a PrivateLink-versus-peering-versus-Transit-Gateway decision procedure, private-only EKS API endpoints, cross-VPC DNS, and a human-access model. Each decision comes with its tradeoffs and the signals that say it is time to revisit it. The guide then walks through failure modes seen in a real, mature single-account platform, all anonymized and generalized: a default CIDR that silently gave dev and prod the same address space, PrivateLink forwarders that became the least-patched hosts in the estate, a KMS key policy that let a non-production deploy role sign with a production fund-moving key, and others. It ends with a prioritized checklist and a maturity roadmap.

1. The problem space

1.1 Why these two decisions are special

Most infrastructure choices can be reversed with a migration that costs a few weeks. These two are different, because both are set when the resource is created and cannot be changed afterwards.

A VPC’s primary CIDR block cannot be changed or removed. You can add secondary blocks, but the primary one stays associated for the VPC’s whole life [8]. VPC peering refuses any overlap across all associated blocks on both sides [7]. So if two VPCs share a primary range, they can never be peered, and no amount of secondary-CIDR work changes that. The only fix is a new VPC and a blue-green rebuild of everything inside it: EKS, databases, brokers, NAT, endpoints.
The AWS account is the only hard isolation boundary AWS offers. IAM policies, permission boundaries, tags and resource policies are all within-account controls. Each one can be misconfigured, and each one is managed by principals who can often change it. AWS’s own guidance treats the account as the unit of isolation for security, billing and quota [1][3]. Moving live workloads between accounts means re-creating KMS keys, re-pointing DNS, migrating databases, and on a blockchain platform, possibly migrating on-chain ownership to new signer addresses. That last step can’t be done by AWS at all.

1.2 The forces at a startup

Force	What it pushes you toward	What it costs you later
Small team (1–3 platform people)	One account, one VPC module with defaults, a hand-rolled VPN	“Admin anywhere is admin everywhere”; access managed by hand
Speed	Copy the dev environment to make prod	Identical CIDRs; dev and prod can’t share a hub
Cost	Single NAT gateway, no Transit Gateway, self-hosted VPN	Fine early on; the real cost is the rework
Partner/auditor pressure (SOC 2, exchange due diligence, custody partners)	Evidence of environment separation, least privilege, audit-log integrity	A single account forces you to prove a negative (“no dev identity can touch prod keys”) one policy at a time
Crypto stakes	KMS-backed signers, hot wallets, relayer EOAs holding gas	One over-broad role means funds can move irreversibly. There is no chargeback on-chain

The crypto row is the one that should change your defaults. On a typical SaaS stack, a compromised dev deploy role can do a lot of damage, but data can be restored and access revoked. On a platform with a KMS key that owns a proxy contract or pays gas, kms:Sign from the wrong principal is a one-way door. CCSS treats key generation, storage and usage controls, and the separation of the people and systems that can use keys, as first-class requirements [19]. Most of those requirements are much easier to meet when the key lives in an account that non-production identities simply cannot reach.

2. Reference design

The design below fits a seed-to-Series-B crypto infrastructure company running EKS, RDS, ElastiCache and a message broker, with KMS-held signing keys. It deliberately avoids anything a two-person team can’t operate.

2.1 Account layout

Account layout: one management account, three organizational units, and a separate account for signing keys. full size ↗

This follows the AWS multi-account whitepaper’s OU structure [1] and the AWS Security Reference Architecture [2], cut down to what a startup actually needs:

Management account. Organizations, consolidated billing, SCPs and RCPs. Nothing else runs here. SCPs do not apply to the management account [5], so anything running in it is outside your guardrails by design.
log-archive. The organization CloudTrail trail and Config delivery, in an S3 bucket with Object Lock. Workload admins cannot delete logs here, because they aren’t admins in this account.
security-tooling. Delegated administrator for GuardDuty, Security Hub, IAM Access Analyzer and Config aggregation.
shared-services. The hub VPC, the human-access gateway, ECR, CI/CD federation roles, the GitOps controller and the observability stack. This account talks to every environment, so treat it as production-tier. A compromise here reaches prod.
dev / prod. One account per environment. Per-engineer sandboxes go under Non-prod with budget actions.
prod-signing (crypto-specific, optional at seed, strongly recommended by Series A). An account that holds only the KMS signing keys, plus the IAM roles that are allowed to call kms:Sign on them. Workloads in prod assume a role into this account, or are named directly as principals in the key policy (KMS key policies accept cross-account principals [17]). The point is that the set of identities able to change the key policy is the people who can administer the signing account, and that is a much smaller set than “anyone with admin in prod”.

2.2 Guardrails as SCPs, not deny patches

In a single account, guardrails end up as inline Deny policies attached to each powerful role, one per role, each one easy to forget on the next role. Under Organizations, the same intent becomes one SCP attached to an OU, and it applies to every principal in every account below it, including that account’s admins [5]. A minimal starting set:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeavingOrgAndTamperingWithAuditTrail",
      "Effect": "Deny",
      "Action": [
        "organizations:LeaveOrganization",
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "config:DeleteConfigurationRecorder",
        "config:StopConfigurationRecorder",
        "guardduty:DeleteDetector",
        "guardduty:DisassociateFromMasterAccount"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyUnusedRegions",
      "Effect": "Deny",
      "NotAction": ["iam:*", "organizations:*", "route53:*", "cloudfront:*",
                    "sts:*", "support:*", "budgets:*", "kms:*", "acm:*"],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": { "aws:RequestedRegion": ["us-east-1", "eu-central-1"] }
      }
    },
    {
      "Sid": "DenyIrreversibleKmsOpsExceptBreakGlass",
      "Effect": "Deny",
      "Action": ["kms:ScheduleKeyDeletion", "kms:DisableKey", "kms:PutKeyPolicy"],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": { "aws:PrincipalArn": "arn:aws:iam::*:role/break-glass-kms" }
      }
    },
    {
      "Sid": "DenyLongLivedHumanCredentials",
      "Effect": "Deny",
      "Action": ["iam:CreateUser", "iam:CreateAccessKey", "iam:CreateLoginProfile"],
      "Resource": "*"
    }
  ]
}

Notes:

The region-deny NotAction list must include the global services you use. Test it in a sandbox OU first, because a wrong list will break Route 53, IAM or CloudFront changes in a way that is confusing to debug.
Attach the KMS statement to the Prod OU, and attach a stricter one to Non-prod that also denies kms:Sign on any key outside the account. Better still, make sure non-prod principals are never named in prod key policies at all.
Resource control policies (RCPs) [6] are the complement: they cap what any principal, including external ones, can do to resources in the account. An RCP can say “no principal outside the organization can call kms:Sign on any key here”. That is exactly the property a signing account needs, and it holds even if someone writes a careless key policy.

2.3 Address plan

Allocate CIDRs centrally, once, before the first VPC exists. Use AWS VPC IPAM [20] if you have Organizations, or a checked-in table if you don’t. Never rely on a network module’s default value.

# network-plan.tf: the single source of truth. Every VPC reads from here.
locals {
  # One /12 per region. /16 per environment VPC. Never overlap, never adjacent by accident.
  vpc_cidrs = {
    "use1-shared"  = "10.64.0.0/16"   # hub: access gateway, endpoints, observability
    "use1-dev"     = "10.65.0.0/16"
    "use1-prod"    = "10.66.0.0/16"
    "use1-signing" = "10.67.0.0/20"   # tiny: only interface endpoints, if any
    # reserved, deliberately unused
    "use1-staging" = "10.68.0.0/16"
    "use1-dr"      = "10.72.0.0/16"
  }
  vpn_client_cidr = "100.96.0.0/20"   # outside RFC1918 entirely: never collides with a home LAN
}

# modules/network/variables.tf
variable "vpc_cidr" {
  type        = string
  description = "Required. Look it up in network-plan.tf. There is intentionally no default."
  validation {
    condition     = can(cidrnetmask(var.vpc_cidr))
    error_message = "vpc_cidr must be a valid CIDR from the central plan."
  }
}

The no default line is the most important line in this guide. Section 4.1 explains why.

Other rules for the plan:

Avoid 10.0.0.0/16, 192.168.0.0/16 and 172.17.0.0/16. Home routers, Docker’s default bridge and many partner networks use them. A VPN client whose LAN is 10.0.0.0/24 will lose routes to a VPC on 10.0.0.0/16.
Size for EKS. With the VPC CNI, every pod takes a VPC IP. A /16 per environment with /19 private subnets per AZ is comfortable. A /20 per environment is what people regret.
Reserve ranges for environments you don’t have yet: staging, DR region, a second prod region, an acquired company’s VPC.

2.4 Network topology

The hub VPC in shared-services, how dev and prod attach to it, and the signing account that is only reachable across an account boundary. full size ↗

Opinions encoded here:

Hub-and-spoke, with the hub in its own account. The hub holds what every environment needs to reach: the access gateway, the GitOps controller, observability ingest. The multi-VPC whitepaper calls this the shared-services pattern [9].
Dev is routed (peering), prod is published (PrivateLink). Even with non-overlapping CIDRs, consider keeping prod on PrivateLink on purpose. A peering or TGW attachment gives the hub a route to every IP in prod, and security groups are then the only filter. PrivateLink exposes exactly the services you publish, one-way, and a compromised hub host cannot scan the prod VPC [10][13]. This works as least privilege at the network layer.
Private-only EKS API endpoints in every environment [14]. The GitOps controller and humans reach them over the hub.
Signing keys outside the workload account, reached by IAM, not by network.

2.5 Publishing a private EKS API over PrivateLink

An EKS cluster with a private-only endpoint exposes its API through EKS-managed ENIs in your subnets. It also gets an EKS-managed private hosted zone that is associated only with the cluster’s own VPC [14]. To consume it from the hub without a route:

# Provider side (prod account): internal NLB whose targets are the EKS API ENI IPs.
resource "aws_lb" "eks_api" {
  name               = "prod-eks-api-pl"
  internal           = true
  load_balancer_type = "network"
  subnets            = module.network.private_subnet_ids
}

resource "aws_vpc_endpoint_service" "eks_api" {
  network_load_balancer_arns = [aws_lb.eks_api.arn]
  acceptance_required        = false
  allowed_principals         = ["arn:aws:iam::${var.shared_services_account_id}:root"]
}

# Consumer side (shared-services account)
resource "aws_vpc_endpoint" "prod_eks_api" {
  vpc_id              = module.hub.vpc_id
  service_name        = var.prod_eks_endpoint_service_name
  vpc_endpoint_type   = "Interface"
  subnet_ids          = module.hub.private_subnet_ids
  security_group_ids  = [aws_security_group.prod_eks_api_endpoint.id]
  private_dns_enabled = false   # would conflict with the EKS-managed zone's name
}

# Override the *real* API hostname inside the hub VPC so kubectl and the GitOps
# controller use the real name and TLS verifies without any skip flags.
resource "aws_route53_zone" "prod_eks_api" {
  name = local.prod_eks_api_host            # e.g. ABCD1234.gr7.us-east-1.eks.amazonaws.com
  vpc { vpc_id = module.hub.vpc_id }
}

resource "aws_route53_record" "prod_eks_api" {
  zone_id = aws_route53_zone.prod_eks_api.zone_id
  name    = local.prod_eks_api_host
  type    = "A"
  ttl     = 60
  records = [for eni in data.aws_network_interface.prod_eks_vpce : eni.private_ip]
}

There are two subtleties here, both covered in Section 5. First, the NLB’s targets are ENI IPs that EKS owns and can rotate, so something has to keep the target group current. Second, any client that connects using the vpce-* hostname must set the TLS server name (tlsClientConfig.serverName in an Argo CD cluster secret [21], tls-server-name in a kubeconfig) instead of disabling verification.

2.6 Publishing databases and brokers: prefer resource endpoints over forwarders

Historically, an RDS instance or an Amazon MQ broker could only be put behind PrivateLink with an NLB, and an NLB only targets IPs or instances. RDS endpoint IPs move on failover. The common workaround was an autoscaling group of small EC2 instances running socat from the NLB port to the database hostname. That works, but it adds a fleet of hand-built hosts on the most sensitive network path you have.

Since late 2024, PrivateLink supports VPC resources directly. A resource gateway in the provider VPC, plus a resource configuration that targets an IP, a domain name, or an AWS resource by ARN (Amazon RDS is the documented example), is shared through AWS RAM [30] and consumed through a resource endpoint [11][12]. Use that for databases and brokers in a new design. If you inherit the forwarder pattern, put it in one module, on a current AMI, with IMDSv2 enforced and a pinned launch-template version. Don’t let it be copied three times.

2.7 DNS across VPCs

There are three mechanisms, and you need to know which problem each one solves:

Need	Mechanism
A name defined by you should resolve inside several VPCs	Private hosted zone associated with each VPC (cross-account association is supported) [16]
A name defined by AWS in another VPC’s PHZ (EKS private API, RDS/`vpce` private DNS) should resolve from a peered VPC or from on-prem/VPN clients	Route 53 Resolver inbound endpoint in the VPC that can see the zone; clients forward those suffixes to its ENI IPs [15]
VPC workloads should resolve a corporate or partner domain	Resolver outbound endpoint + forwarding rules, shared org-wide via RAM [15]

For VPN clients, configure split DNS: forward only the internal suffixes (internal.example.dev, eks.amazonaws.com, rds.amazonaws.com, vpce.amazonaws.com) to the inbound endpoint IPs, and leave everything else on the client’s normal resolver.

2.8 Human access

Three ways a human reaches private resources: an identity-aware tunnel, a plain SSM session, and a hardened ops host. full size ↗

Three paths, each for a different job:

Daily access: an identity-aware overlay. This means a managed WireGuard-based mesh, or AWS Verified Access for HTTP apps. Peers authenticate against your IdP, access follows group membership, and keys rotate automatically.
Fallback: SSM Session Manager port forwarding [22]. It needs no inbound port, it is authorized by IAM (ssm:StartSession plus an MFA condition), and it is logged in CloudTrail. It is the one path that still works when the VPN, the IdP or the access gateway is down, so keep it working even if nobody uses it daily.
```
aws ssm start-session --target "$BASTION_ID" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["prod-db.proxy-xyz.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["15432"]}'
```
Privileged one-offs: a jailed ops host. This is a private instance whose SSM session documents start a restricted dispatcher, not a shell. Credentials are minted on the box: 15-minute RDS IAM tokens, and instance-role access to KMS. Every mutation writes an audit record. The goal is that production credentials never reach a laptop.

3. Decisions and tradeoffs

D1. Single account vs AWS Organizations

	Single account	Organizations, multi-account
Setup cost	Zero	1–3 engineer-weeks with Control Tower; more if hand-rolled
Isolation	IAM only. Every guardrail is a policy someone can edit	Account boundary + SCPs/RCPs that account admins can’t override [5][6]
Blast radius of a leaked admin role	Everything, every environment	One account
Audit-log integrity	Admins can stop or modify trails unless denied role by role	Org trail in log-archive; workload admins can’t touch it
Billing visibility	Tags, which are always incomplete	Per account, exact
Quotas	Shared: a dev load test can exhaust prod’s ENI or EIP quota	Per account
Ongoing cost	—	Cross-account plumbing: RAM shares, cross-account roles, per-account baselines

Upsides of single-account early: speed, one console, no cross-account IAM to debug. For a pre-product team with no keys holding value, it is defensible.

The downside is structural, not incidental. An independent review of one mature single-account platform found that most of its critical and high security findings had a single root cause: “admin anywhere is admin everywhere”. A non-production deploy role could sign with a production key. A PR-triggered CI plan role could read every secret in the account, including a funded wallet’s key left behind by a decommissioned service. One container-push role could push and delete images for every repository. Each finding had a targeted mitigation (deny statements, scoped ARNs, removed actions), and every mitigation was a patch on a model that would keep producing new findings. Per-environment accounts would have removed whole classes of them at once.

Revisit signals: the first KMS key that controls funds; the first auditor or custody-partner questionnaire that asks about environment separation; the first time you write a Deny policy whose purpose is “stop dev from touching prod”; a second engineer with admin. For a crypto company, at least one of these happens before Series A.

D2. How many accounts

Start with management, log-archive, security-tooling, shared-services, dev, prod: six. Add prod-signing as soon as you have keys that control funds or contract ownership. Add per-engineer sandboxes once people start experimenting in dev. Add a network account only when you run a Transit Gateway or IPAM pools that several teams administer.

Don’t create an account per microservice at startup scale. The per-account baseline (Config, GuardDuty, SSO assignments, budget, IAM roles for CI) is real work, and accounts are hard to delete. A closed account stays suspended for a 90-day post-closure period before it is gone, so treat account creation as cheap and deletion as slow.

D3. Control Tower vs hand-rolled Organizations

AWS Control Tower [4] sets up the landing zone (management, log-archive and audit accounts), the org trail, IAM Identity Center, and a catalog of preventive (SCP) and detective (Config) controls. Account Factory, optionally with Account Factory for Terraform, vends new accounts with a baseline. Upside: opinionated, supported, auditor-recognizable. Downside: it owns some OUs and SCPs, and drift from its expectations has to be repaired through its console; some teams find its CloudFormation-StackSet mechanics opaque.
Hand-rolled (Organizations + SCPs + Identity Center in Terraform). Upside: everything is in your repo and reviewed like any other change. Downside: you write the baseline yourself, and the first version will be missing something.

Recommendation: Control Tower for a team with no prior multi-account experience. Hand-rolled if you already run Terraform rigorously and want a single source of truth. Either way, IAM Identity Center replaces IAM users for humans. In practice, the identity-aware access work in D6 often stalls because “we’d federate through Identity Center, but that needs an Organization first.”

The prerequisite nobody budgets for: root. Creating an Organization requires the root user (or an admin) of the account that will become the management account. Usually you create a new management account and invite the existing account into it. The existing account then becomes a member, which is fine to start with, though its workloads stay mixed until you split them out. One team found its multi-account plan parked indefinitely because no current engineer had root or management access to anything. Fix root custody on day one: hardware MFA, credentials in a sealed two-person process, a documented recovery email that isn’t a departed founder’s mailbox.

D4. CIDR planning

Option	Upside	Downside	When
Module default (`10.0.0.0/16` everywhere)	No thought needed	Every environment collides; peering and TGW impossible later	Never
Hand-maintained table in the repo	Free, reviewable	Relies on discipline; no enforcement across accounts	Pre-Organizations
VPC IPAM with pools per region and OU [20]	Allocation is enforced; overlaps detected; works across accounts	Costs per active IP; another service to learn	Once you have Organizations

Revisit signal: the first time anyone says “we’ll just peer these two VPCs”. Check for overlap before that conversation, not during the apply.

D5. Connecting VPCs: peering vs Transit Gateway vs PrivateLink

	VPC peering	Transit Gateway	PrivateLink (endpoint services / resource endpoints)
What’s shared	Full L3 reachability between two VPCs	Full L3 between all attachments, subject to TGW route tables	One service (NLB/GWLB-fronted) or one resource, one direction
Overlapping CIDRs	Not allowed [7]	Attachments work, but you can’t route between overlapping ranges without NAT tricks (private NAT gateways [23])	Irrelevant: the consumer sees an ENI in its own CIDR
Transitive routing	No	Yes (hub)	N/A
Scale	O(n²) connections; route-table entries per peer	Thousands of attachments [18]	One endpoint per service per consumer VPC
Cost (us-east-1, check current pricing)	No hourly charge; standard data transfer	Per attachment-hour + per-GB processing [18]	Per endpoint per AZ-hour + per-GB [24]
Security posture	Every IP reachable, SGs the only filter	Same, with route-table segmentation	Consumer can reach only what’s published; provider can’t initiate back
Ops complexity	Low	Medium (route tables, propagation, appliance VPCs)	Medium per service, low per consumer

Decision procedure:

Do the CIDRs overlap? If yes, PrivateLink is your only clean option short of a rebuild. Plan the rebuild of the lower-risk environment anyway (see 5.1), but don’t put prod in the critical path of that rebuild.
Do you need many-to-many reachability across more than about 4 VPCs, or on-prem/partner connectivity? Use a Transit Gateway in a network account, shared by RAM [30].
Do you need a few VPCs (hub plus two to three spokes) with no transitive needs? Use peering for non-prod.
Is the target prod and the consumer a shared or tooling VPC? Prefer PrivateLink even when routing is possible: least privilege at L3.

Revisit signals: a fourth spoke; a second region; a partner asking for private connectivity (PrivateLink endpoint services are the standard way to offer a private API to customers [13]); more than about 10 PrivateLink endpoints in the hub, at which point per-endpoint-hour costs and per-service SG management start to exceed the cost of a TGW.

D6. Human access: VPN vs SSM vs bastions vs identity-aware proxies

Option	Identity at connect time	Per-resource policy	HA	Rough cost (small team)	Ops burden
Self-hosted WireGuard on EC2 [25]	Possession of a private key	No (network-level)	Single instance unless you build it	One small instance	Peer lifecycle by hand; manual revocation
AWS Client VPN + SAML	IdP via SAML	No	Managed	Per connection-hour + per subnet association	Low
Managed overlay mesh (WireGuard-based)	IdP SSO, device posture, key expiry	Network ACLs as code	Multiple subnet routers fail over	Per seat, often under $100/month for a small team	Low; adds a vendor control plane
AWS Verified Access [26]	IdP (OIDC) + optional device trust, evaluated per request	Yes, per app/endpoint	Managed	Per app-hour or endpoint-hour; easily several hundred $/month for a handful of apps	Medium; one endpoint per resource
Teleport / access proxy	IdP (SSO in paid tiers)	Yes, per resource, with session recording	You run it	License or ops time	High for a small team
SSM Session Manager port forwarding [22]	IAM + MFA, every session in CloudTrail	Per target instance (IAM)	Managed	Effectively free	Low, but awkward for daily use

Recommendation by stage:

Seed: self-hosted WireGuard is acceptable if you understand what it doesn’t give you (see 5.8): no identity per session, no MFA at the tunnel, and a leaked .conf is full network access until someone removes the peer. Keep SSM port forwarding as the documented fallback.
Series A: move to a managed mesh with IdP group sync and automatic deprovisioning. Without deprovisioning, offboarding is still a manual step in a second console, which is half of the problem you were solving. Run two subnet routers in private subnets across AZs, with SNAT on, so downstream security groups don’t change.
Scale / regulated: add Verified Access in front of HTTP admin UIs (Grafana, Argo CD, BI tools) for per-request policy, which follows the BeyondCorp and NIST zero-trust models [27][28]. Don’t put Verified Access in front of every TCP database endpoint; that’s where per-endpoint cost grows fastest for the least added benefit over a mesh.

At every stage, keep SSM as break-glass, because it is the only path that depends on neither your VPN nor your IdP.

D7. Private-only EKS API endpoints

Run the EKS API endpoint private-only in every environment [14]. The cost is that every client (humans, CI, the GitOps controller) now needs a network path and DNS for the private endpoint. That is exactly what the hub provides. A public endpoint with a CIDR allowlist is a reasonable stopgap, but CIDR allowlists drift, and “temporary” 0.0.0.0/0 entries are among the most common audit findings.

Revisit signal: none. Once private, stay private. If CI needs access, run CI runners in the hub or use SSM.

D8. Where signing keys live

Option	Who can change the key policy	Who can `Sign`	Notes
Workload account, key policy starts with `root: kms:*`	Any IAM principal the account grants `kms:PutKeyPolicy`	Any admin in the account, unless an explicit Deny exists	The AWS default key policy delegates to IAM [17], so “admin” means “signer”
Workload account, key policy with explicit `Deny Sign` for unlisted principals	Still any admin with `PutKeyPolicy` (unless SCP-denied)	Only listed roles	Good mitigation; but every new key must remember it
Dedicated signing account + RCP	Only signing-account admins (a very small set)	Only named cross-account roles	Structural; recommended for keys holding value

Key policy for a signer, using principal and network-path conditions instead of source IP:

{
  "Sid": "DenySignByAnyoneButTheSigner",
  "Effect": "Deny",
  "Principal": "*",
  "Action": ["kms:Sign"],
  "Resource": "*",
  "Condition": {
    "ArnNotEquals": {
      "aws:PrincipalArn": "arn:aws:iam::111111111111:role/prod-signer-svc"
    }
  }
},
{
  "Sid": "DenySignOffTheExpectedPath",
  "Effect": "Deny",
  "Principal": "*",
  "Action": ["kms:Sign"],
  "Resource": "*",
  "Condition": {
    "StringNotEquals": { "aws:SourceVpce": "vpce-0abc123prodkms" }
  }
}

Scope these to kms:Sign only. Do not put kms:GetPublicKey behind the same deny (see 5.6). Requiring a KMS interface endpoint and conditioning on aws:SourceVpce [29] is more robust than conditioning on NAT EIPs. NAT EIPs change when you rebuild networking, and an IP condition silently stops matching anything once traffic moves to a VPC endpoint (where aws:SourceIp isn’t populated).

4. The root-cause lesson: defaults are decisions

4.1 The default CIDR that froze the topology

The most expensive line in one platform’s network code was an innocent default in a shared VPC module, "10.x.0.0/16". Neither the dev root module nor the prod root module overrode it, so the two VPCs got the same primary CIDR. Nobody noticed for a long time, because separate VPCs work fine in isolation.

It came to light when the team built a shared VPN/tooling VPC and tried to peer it with both environments. A route table can hold only one route per destination CIDR, so the hub could peer with dev or prod, never both. The Terraform ended up encoding the workaround: a cidr_overlap local, a “preferred environment” variable defaulting to dev, and the prod peering resource permanently at count = 0. Prod became reachable only through a set of PrivateLink endpoints for its EKS API, its databases, its message broker and its admin ingress.

PrivateLink was a fine outcome for prod (see D5, point 4), and the team rightly wrote “don’t ‘fix’ it into a peering” in their docs. But the choice wasn’t a choice; it was the only option the CIDR math allowed. The landmine stays armed for the next person who tries to add a TGW attachment spanning both VPCs, or any tool that assumes environment CIDRs are unique. And the fix is a blue-green rebuild of the whole dev environment: a new VPC, recreated EKS and bastion, RDS snapshot/restore, a broker definitions export/import, new peering, a new Resolver inbound endpoint, every hardcoded CIDR literal found by grep (the ones a variable rename misses), and reissued VPN client configs.

Lesson: a network module’s vpc_cidr must have no default. Allocate from a central plan (2.3). Add a CI check that fails if any two VPCs in the plan overlap.

# ci/check_cidrs.py: fail the build on any overlap in the central plan
import ipaddress, json, itertools, sys
plan = json.load(open("network-plan.json"))
nets = {k: ipaddress.ip_network(v) for k, v in plan.items()}
bad = [(a, b) for a, b in itertools.combinations(nets, 2) if nets[a].overlaps(nets[b])]
for a, b in bad:
    print(f"OVERLAP: {a} {nets[a]} <-> {b} {nets[b]}")
sys.exit(1 if bad else 0)

5. Failure modes and field lessons

Each of these comes from real platforms, generalized. Most share one shape: the plan was clean, and the failure was in a transition, a fallback path, or an assumption nobody wrote down.

5.1 You can’t add your way out of a CIDR overlap

Teams often try to fix an overlap by adding a secondary CIDR, moving workloads onto it, and then peering. That doesn’t work. Peering checks overlap across all associated blocks, and the primary can’t be disassociated [7][8]. The only real fix is a new VPC.

When you do rebuild, rebuild the lower-risk environment (dev), and do it in phases: (0) choose the structure, either a parallel root module or a feature flag in the existing one. A parallel root has cleaner blast-radius isolation; a flag means singleton names (cluster name, RDS identifiers) collide while both exist. (1) Stand up the network, peering and Resolver endpoint first. (2) Recreate stateless pieces. (3) Migrate stateful data and keep the old copies. (4) Cut over. (5) Decommission after a soak period you choose deliberately. This is also a good rehearsal for full-environment DR.

5.2 PrivateLink forwarders become the least-maintained hosts you own

The NLB-plus-socat-on-EC2 pattern for publishing RDS and brokers over PrivateLink (2.6) was copied three times in one estate, once per database or broker. All three copies pinned an AMI filter for an operating system that had reached end of life. All three were the only launch templates in the account that didn’t require IMDSv2. Every other instance had been hardened by a sweep that searched for aws_instance and the standard launch-template modules, and these didn’t match either search. They were the only VPN path to the prod databases.

Lessons: forwarders on a privileged path need the same baseline as everything else. Copy-paste is where shared weaknesses come from, so one module, not three. Where possible, replace forwarders with PrivateLink resource endpoints [11].

5.3 The EKS-API-over-PrivateLink pattern has three sharp edges

Target drift. The provider NLB targets the EKS-managed API ENIs. Those IPs are stable during normal operation but change when the cluster is recreated, and they can rotate. One team keyed target attachments by ordinal (to keep Terraform state stable), added a sentinel tied to the cluster’s creation timestamp so attachments are replaced when the cluster is, and ran a scheduled Lambda to reconcile the IPs between applies.
Private DNS conflicts. Setting private_dns_enabled = true on the consumer endpoint tries to create a zone that clashes with the EKS-managed one. Leave it off, and add your own PHZ in the hub that overrides the real API hostname with the endpoint ENI IPs (2.5). Then aws eks update-kubeconfig works unchanged and TLS validates.
Hostname mismatch. Clients that use the vpce-* DNS name must set the TLS server name. Tunnel scripts that set insecure_skip_tls_verify “because it was easier” turn up in audits. tls-server-name has been supported by kubeconfig for years.

Related: Terraform data sources that read the cluster endpoint from another state’s outputs can pick up a tunnel address (https://127.0.0.1:6443) if that state was last applied through a tunnel wrapper. Read the endpoint from the EKS API directly (data "aws_eks_cluster"), not from remote state.

5.4 TLS identity vs routing identity

A managed broker published over PrivateLink has a certificate issued for its broker hostname, not for the vpce-* name. Point an AMQPS client at the endpoint hostname and verification fails. The working pattern is to treat the endpoint hostname as the routing target only: map the broker hostname to the endpoint ENI IP (a PHZ in the hub, or /etc/hosts locally) and connect using the broker name. Postgres behind RDS Proxy is often fine with the endpoint hostname, depending on sslmode. So two endpoints in the same file are used differently, and the Terraform output descriptions are where that has to be written down.

5.5 Routing gotchas with a gateway instance in a public subnet

A VPN gateway instance with an Elastic IP sits in a public subnet. Its forwarded traffic leaves through the public route table, which by default has no route to peered VPCs. Packets bound for the peer match 0.0.0.0/0 → igw and are silently dropped. Add the peering route to the public route table too, not only the private ones.

Also note that hub-side Terraform that writes routes into spoke route tables (looked up from the spoke’s remote state) creates a cross-state write. A spoke apply that recreates its route tables orphans the hub’s routes until the hub applies again. Prefer each side owning its own routes, with IDs passed through SSM parameters rather than remote-state reads in both directions. Remote-state graphs with reverse edges become cycles that make a from-scratch rebuild need hand-staged -target applies.

5.6 IP-based KMS conditions break disaster recovery

One team restricted kms:Sign on production keys to the production NAT gateway EIPs. Reasonable, except that the same Deny statement covered kms:GetPublicKey. That made it impossible to read the public key, and therefore to derive the on-chain address, from anywhere outside the production network. Months later, a DR inventory found that the addresses of roughly half the production signing keys were simply unknown to the team. Recovering them meant running a command from inside the cluster, one key at a time.

Lessons:

Condition only the dangerous action (kms:Sign), not read-only ones.
Record each key’s public key and derived address at creation time, as a Terraform output or in your asset inventory.
Prefer aws:SourceVpce and principal conditions over aws:SourceIp [29].
Remember that KMS key deletion is a 7–30 day pending window, and prevent_destroy on the Terraform resource is cheap insurance.

5.7 Single-account blast radius, by example

Generalized from one platform’s review, which ranked this as its only critical finding:

The signing-key module started its key policy from the AWS default (root: kms:*), which delegates authority to IAM [17]. Most production keys later gained an explicit “deny signing by unapproved principals” statement. One did not: a privileged “owner” key used for contract administration, which several developers also used locally under different identities. Adding the principal restriction would have broken those workflows, so it was deferred.
The account had inline guardrail policies that denied key deletion and policy changes to deploy roles, but not kms:Sign.
The non-production deploy role, which had no approval gate, held AdministratorAccess in the same account. So by IAM evaluation it could sign with the production owner key.
Separately, one fee-payer key was shared by dev and prod and owned by the dev Terraform state.

None of this was negligence. Each step was a locally reasonable shortcut. But every fix inside a single account is another Deny to remember. In a dedicated signing account with an RCP forbidding external kms:Sign, none of these paths exist. The short-term mitigation order is: add kms:Sign to the deny list on every non-prod role; inventory every identity that actually signs with each privileged key before adding a principal restriction (or you break contract deploys mid-release); split shared keys into environment-owned keys, which needs an on-chain cutover.

5.8 Static VPN peers: what you don’t get

A self-hosted WireGuard gateway is simple and fast. What it doesn’t give you:

No identity past the tunnel. The gateway SNATs client traffic, so every downstream service sees the gateway’s IP. Grafana, kubectl and psql logs can’t tell you who came through the tunnel, only that someone did. API calls still carry IAM identity, so CloudTrail and EKS audit logs are intact. Network-level forensics are not.
No MFA at the tunnel. Holding the .conf is sufficient. Phone configs travel as QR codes, so treat them as secrets and deliver them only through a password manager.
Revocation is a Terraform change, and “who has access” means grepping peer blocks. Peers added live with wg set drift from the declared list.
SPOF. EC2 auto-recovery alarms restart the same instance. That’s recovery, not HA. If it is the only human path, keep SSM working.
Vestigial rules. Because of SNAT, security-group rules that admit the tunnel client CIDR are almost always dead. The real admission is the hub VPC CIDR. Clean them up after migrating to a new access system, never before.

5.9 DNS: three ways to be “connected but broken”

Peered but unresolvable. EKS’s private-endpoint PHZ is associated only with the cluster VPC. A peered hub has routes but can’t resolve the name until you associate the zone with the hub VPC, or forward to a Resolver inbound endpoint in the spoke.
Mobile clients ignore PostUp hooks. WireGuard configs that set up split DNS with PostUp scripts work on laptops and do nothing on iOS or Android. Use the DNS = line with trailing search domains, which the mobile apps treat as split-DNS match domains, or accept that phones only reach web UIs.
Public zones returning private IPs. Publishing grafana.internal.example.dev → 10.64.x.x in a public zone means any resolver can answer, which is convenient. But some home routers and DNS-rebinding protections drop RFC1918 answers for public names, so the site “works at the office, not at home”. Either serve internal names through the VPN’s resolver (split DNS) or document the rebinding workaround. Two documents in the same repo disagreeing about whether VPN DNS is needed is a sign this edge case has already bitten someone.

Older client configs issued before a DNS change won’t have the new resolver block. Reissue the config instead of debugging per laptop.

5.10 “Internet-facing by construction, closed by an empty list”

A shared database module supported an optional public NLB and forwarder (for a third-party tool), with publicly_accessible = true on the instance and the proxy. In dev it was enabled, in public subnets, and the only thing keeping it closed was an empty ingress CIDR list. One well-meaning PR adding an office IP would have opened it. Lesson: make the unsafe thing structurally impossible (private subnets, both flags false, feature removed from the module) instead of guarded by configuration.

5.11 Dead config that still does something

A client_vpn_enabled = true flag survived the removal of the Client VPN endpoint it once configured, and it still opened a security-group ingress rule and added a route. An unreferenced load-balancer security group open to 0.0.0.0/0 sat in the hub. Neither was exploitable on its own. Both are what an auditor finds first, and both make the next engineer’s mental model wrong. Grep for flags whose consumer is gone. Treat “temporary” as a tracked list with owners and dates.

5.12 Decommissioning an environment in a shared account leaves residue

When one team decommissioned its staging environment, two KMS signing keys couldn’t be destroyed. They held funds or protected addresses that were still referenced. The environment’s Terraform root therefore had to survive as a shell managing only those keys, removed from CI so nothing could apply it. Applying it would have recreated the whole environment, databases included, on shared instances. Separately, a decommissioned service’s secret, containing a still-funded wallet key, stayed readable by CI roles long after its compute was deleted.

Lessons: sweep funds before deleting compute. Inventory every key and secret per environment. In a multi-account world, put staging in its own account so decommissioning means “move the keys you must keep to the signing account, then suspend the account”. That is a far cleaner end state than a half-deleted Terraform root.

5.13 Shared-services is production

The hub account hosts the GitOps controller (which deploys to prod), the access gateway (the only path to prod for humans) and observability (which pages for prod). Treat its changes with prod rigour: the same approval gates, the same guardrails, the same SCP OU. A common anti-pattern is to put it in the “non-prod” mental bucket because it isn’t customer-facing.

6. Best-practice checklist

Day 1 (seed, or the day you read this)

Root user of every account: hardware MFA, no access keys, documented two-person recovery, recovery email on a shared alias.
Decide the account model now, even if you defer it. If you’ll have keys that control funds within 12 months, create the Organization and management account now; it’s an afternoon.
Central CIDR plan checked into the repo; no default on vpc_cidr; CI overlap check.
Avoid 10.0.0.0/16, 192.168.0.0/16, 172.17.0.0/16; VPN client CIDR in 100.64.0.0/10 or another non-RFC1918-colliding range.
EKS API endpoints private-only.
No IAM users for humans or CI. Use Identity Center (or MFA-enforced groups as an interim) and GitHub OIDC roles scoped to repo:<org>/<repo>:ref:refs/heads/main for anything that applies.
SSM Session Manager port forwarding documented and tested as the break-glass path.
Every signing key: an explicit Deny kms:Sign for unlisted principals; public key and address recorded at creation; prevent_destroy.
CloudTrail in every region, with log-file integrity validation, delivered somewhere workload admins can’t delete.

First year (Series A)

AWS Organizations with management, log-archive, security-tooling, shared-services, dev and prod accounts; Control Tower or Terraform-managed equivalent.
SCPs: deny leaving the org, deny trail/Config/GuardDuty tampering, deny unused regions, deny IAM user creation, deny irreversible KMS operations outside a break-glass role.
Dedicated prod-signing account; RCP forbidding external kms:Sign; workloads reach it by named cross-account role.
Hub-and-spoke with prod on PrivateLink; databases and brokers via resource endpoints rather than forwarder fleets.
Resolver inbound endpoints where AWS-owned private names must resolve from the hub or VPN; split DNS on clients.
Identity-aware human access (managed mesh with IdP group sync and automatic deprovisioning), two routers across AZs; WireGuard peers retired.
A jailed ops host for privileged one-offs so prod credentials never land on laptops.
CI plan roles split: PR plans get no secrets and only non-prod state; prod plans run from main only.
VPC IPAM pools per region/OU.

At scale (Series B+)

Network account with Transit Gateway once spokes exceed about 4 or you add a region or partner connectivity; TGW route tables segmenting prod from non-prod.
Verified Access in front of HTTP admin apps for per-request, device-aware policy.
Per-engineer sandbox accounts with budget actions.
Second-region DR design using reserved CIDRs; drill an environment rebuild end to end.
Periodic “claim audit” of network and access docs against the code. Drift in load-bearing claims (“prod is peered”, “the VPN needs no DNS”) costs the most during incidents.

7. Maturity roadmap

Stage	Accounts	Network	Human access	Keys	Signals to move on
0 — Prototype	1 account, root locked	One VPC from the central plan, private EKS endpoint	SSM port forwarding	Test keys only	First key that controls real value
1 — Seed / first mainnet	Organization created; existing account becomes a member; log-archive account	Hub VPC + dev/prod VPCs, non-overlapping; dev peered, prod over PrivateLink	Self-hosted WireGuard + SSM fallback	Signing keys with principal Deny; addresses recorded	Second admin; partner security questionnaire; first “dev mustn’t touch prod” Deny
2 — Series A	management / log-archive / security / shared-services / dev / prod / prod-signing; SCPs + RCPs; Identity Center	IPAM; resource endpoints for DBs; Resolver endpoints	Managed identity-aware mesh, 2 AZ routers; jailed ops host	Keys in prod-signing behind RCP	More than 4 spokes; second region; customer asks for private connectivity
3 — Scale	Add network account, sandboxes, per-region DR accounts	Transit Gateway with segmented route tables; PrivateLink endpoint services offered to customers	Verified Access for HTTP apps; mesh for TCP; SSM break-glass	Per-service signing roles; quorum or approval workflows on privileged keys	Regulatory regime (e.g. custody licensing) dictates the next controls

The order matters more than the speed. Each stage makes the next one cheap: a central CIDR plan makes TGW possible, an Organization makes Identity Center and SCPs possible, a signing account makes RCPs meaningful. Skipping a stage means paying for it later with a rebuild.

8. References

AWS. Organizing Your AWS Environment Using Multiple Accounts (whitepaper). https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/organizing-your-aws-environment.html
AWS Prescriptive Guidance. AWS Security Reference Architecture (AWS SRA). https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/welcome.html and The AWS Security Reference Architecture: https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/architecture.html
AWS Well-Architected Framework, Security Pillar. SEC01-BP01 Separate workloads using accounts. https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/sec_securely_operate_multi_accounts.html (pillar overview: https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html)
AWS. What Is AWS Control Tower? https://docs.aws.amazon.com/controltower/latest/userguide/what-is-control-tower.html
AWS Organizations. Service control policies (SCPs). https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html
AWS Organizations. Resource control policies (RCPs). https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html
Amazon VPC. How VPC peering connections work (overlapping CIDR and transitive-routing limitations). https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-basics.html
Amazon VPC. VPC CIDR blocks (primary and secondary CIDR associations). https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html
AWS. Building a Scalable and Secure Multi-VPC AWS Network Infrastructure (whitepaper). https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/welcome.html
AWS PrivateLink. AWS PrivateLink concepts. https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html
AWS PrivateLink. Access VPC resources through AWS PrivateLink. https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html
AWS PrivateLink. Resource configuration for VPC resources. https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html
AWS. Securely Access Services Over AWS PrivateLink (whitepaper). https://docs.aws.amazon.com/whitepapers/latest/aws-privatelink/aws-privatelink.html and Share your services through AWS PrivateLink: https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html
Amazon EKS. Cluster API server endpoint. https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
Amazon Route 53. What is Route 53 VPC Resolver? (inbound and outbound endpoints, forwarding rules). https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html
Amazon Route 53. Working with private hosted zones. https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-private.html
AWS KMS. Key policies in AWS KMS. https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html
Amazon VPC. What is AWS Transit Gateway for Amazon VPC? https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html; pricing: https://aws.amazon.com/transit-gateway/pricing/
CryptoCurrency Certification Consortium (C4). CryptoCurrency Security Standard (CCSS). https://cryptoconsortium.org/standards-2/
Amazon VPC IPAM. What is IPAM? https://docs.aws.amazon.com/vpc/latest/ipam/what-it-is-ipam.html
Argo CD. Declarative Setup: Clusters (tlsClientConfig.serverName). https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
AWS Systems Manager. Start a session (including port forwarding to remote hosts). https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html
Amazon VPC. NAT gateway use cases (“Enable communication between overlapping networks”). https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html
AWS. AWS PrivateLink Pricing. https://aws.amazon.com/privatelink/pricing/
J. A. Donenfeld. WireGuard: Next Generation Kernel Network Tunnel (NDSS 2017). https://www.wireguard.com/papers/wireguard.pdf
AWS. What is AWS Verified Access? https://docs.aws.amazon.com/verified-access/latest/ug/what-is-verified-access.html
NIST. SP 800-207, Zero Trust Architecture. https://csrc.nist.gov/pubs/sp/800/207/final
R. Ward, B. Beyer. BeyondCorp: A New Approach to Enterprise Security (Google Research). https://research.google/pubs/beyondcorp-a-new-approach-to-enterprise-security/
AWS KMS. AWS KMS condition keys (and global condition keys such as aws:SourceVpce). https://docs.aws.amazon.com/kms/latest/developerguide/conditions-kms.html
AWS Resource Access Manager. What is AWS RAM? https://docs.aws.amazon.com/ram/latest/userguide/what-is.html