Building on AWS · guide 12 of 13

The Edge and the Data Tier: Load Balancers, DNS, Certificates, Postgres, Redis, RabbitMQ and RPC Proxies

7426 words · about 38 minutes · all guides

Abstract. This guide covers the two ends of a request’s path that small teams most often get subtly wrong. One end is the edge: the load balancer, the ingress controller, health checks, DNS, DNSSEC, CAA and certificates. The other is the managed data tier: RDS Postgres with RDS Proxy, ElastiCache Valkey/Redis, Amazon MQ for RabbitMQ, and, for blockchain companies, a self-hosted RPC proxy that sits between your services and node providers. It is written for senior DevOps, SRE and platform engineers at seed-to-Series-B startups on AWS, especially crypto infrastructure teams running relayers, indexers, deposit flows and signing services. You will get an opinionated reference design with config, the tradeoffs behind each decision and the signals that say it is time to revisit, a set of anonymized failure stories from production, a prioritized checklist and a maturity roadmap.

Most failures in this guide have one thing in common. The system kept reporting healthy while it did the wrong thing. A target group showed every pod healthy and blackholed traffic. A DNS zone resolved from inside the VPC and returned SERVFAIL for every customer. A rate limiter applied no limit and logged at a level production never prints. A cache held primary data and threw it away. The managed services did what they were told, and what they were told was not what the team believed.

1. The problem space

1.1 Forces on a startup

Small team, wide surface. Two to five platform engineers own the VPC, the clusters, DNS, mail authentication, three or four data stores and the vendor accounts. Nobody specializes. A gotcha that lives only in one person’s head will hit someone else at 3 a.m.
Managed services hide mechanisms. RDS Proxy, ElastiCache, Amazon MQ and the AWS Load Balancer Controller remove most operational toil. They also hide the defaults that bite: an idle-client timeout, an eviction policy, a CAA check that follows CNAMEs, a controller that reconciles over a manual change.
Speed versus safety. Edge and data changes are the hardest to roll back. A DNS negative answer is cached for the SOA minimum. A wiped cache cannot be refilled. A broker cannot be replaced without losing its queues. “Ship and watch” works poorly when the blast radius includes every customer at once.
Cost pressure that pushes the wrong way. A single-node cache.t4g.micro costs about a tenth of a Multi-AZ pair. Skipping RDS Proxy saves roughly $0.015 per vCPU-hour. Dev environments get one NAT Gateway. These are reasonable savings until the wrong store is the one that got the discount.
Partner and auditor pressure. Exchanges, custodians and enterprise customers will ask about WAF, TLS policy, DMARC, DNSSEC and encryption at rest. An answer like “WAF is planned, but our edge cannot attach it” is common, and you want to find it out yourself before a questionnaire does.

1.2 Crypto-specific stakes

Data loss means stuck funds, not just a degraded experience. One deposit flow lost its routing and session records when a cache node was replaced. Funds that had already confirmed on-chain could not be attributed to accounts. Recovering them took a day of manual reconciliation, and some signatures could never be rebuilt.
Turning a transient error into a terminal one. A node vendor’s 429 is transient. A code path that marks a deposit FAILED with a terminal error code on that 429 makes it permanent. Your RPC proxy’s rate limits, retries and failover policy are part of your fund-safety story.
Vendor quotas are hard ceilings. RPC providers bill per request or per compute unit, with plan caps that return 429 max usage reached. The cap applies to the whole account, not per pod. If your limiter is per pod, your budget is fiction.
The edge is the attack surface. Public RPC endpoints, WebSocket relays and deposit widgets attract scraping, credential stuffing and cost-amplification attacks. Rate limiting at the edge has to mean what it says.

2. Reference design

2.1 Topology

The request path: DNS, the three edge entry points, the router in front of the services, and the data tier behind them. full size ↗

2.2 The public edge: NLB + Traefik

The default for an EKS shop that wants one ingress controller everywhere is an internet-facing Network Load Balancer created by the AWS Load Balancer Controller from the ingress controller’s Service, with IP targets (pods are registered directly, skipping kube-proxy) [9][10].

# Traefik Helm values (excerpt). Everything here is load-bearing.
service:
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "websecure"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:...:certificate/..."
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: "ELBSecurityPolicy-TLS13-1-2-2021-06"
    # Health check on a DEDICATED ping port, not the traffic port (see §4.2)
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "HTTP"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8082"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/ping"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "2"
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: >-
      preserve_client_ip.enabled=true,
      deregistration_delay.timeout_seconds=30,
      deregistration_delay.connection_termination.enabled=true
  websecureTargetPort: 8443          # a non-privileged port; pods run as non-root

ports: { web: 80, websecure: 8443, ping: 8082 }
args:
  - "--entrypoints.websecure.address=:8443"
  - "--entrypoints.ping.address=:8082"
  - "--ping"
  - "--ping.entrypoint=ping"
  # Redirect only once TLS reliably lands on websecure (see §4.2, step 3)
  - "--entrypoints.web.http.redirections.entrypoint.to=:443"
  - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
  - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
terminationGracePeriodSeconds: 60   # must exceed deregistration delay

Notes that are not obvious from the documentation:

TLS terminates at the NLB, so Traefik sees plain HTTP on its websecure entrypoint. The redirect target is the literal :443, not the entrypoint name. If you name the entrypoint, Traefik builds https://host:8443 from its address.
The controller owns the listener certificate. A manual aws elbv2 modify-listener is reverted on the next reconcile. Change certificates only through the Service annotation.
Deregistration delay must be shorter than the pod’s termination grace period, and a preStop sleep longer than the health-check failure window lets the NLB drain the pod before Traefik stops accepting connections.
Name the NLB A-only. These NLBs are ipv4, so a copied AAAA record resolves to nothing. Only CloudFront names should be dual-stack.
Cross-zone load balancing is off by default on NLBs. A newly created NLB can refuse connections on one AZ’s IP for a few minutes after it reports active, even while every target shows healthy. Test every address the name returns before you diagnose anything.

2.3 Internal tools: internal ALB + OIDC

Grafana, ArgoCD, a BI tool and back-office UIs go behind an internal ALB in private subnets, reachable only over the VPN or PrivateLink, with ALB-native OIDC authentication through Cognito, federated to your Google/Okta IdP [11]. An aws_lb with internal = true gets only private IPs, so the internet cannot route to it at the network layer. Authentication is the second layer, not the only one.

Use TargetGroupBinding when Terraform owns the ALB and NLB (so DNS, certs, WAF and access logs live with the rest of your edge) but the targets are Kubernetes pods [12]:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata: { name: grafana, namespace: observability }
spec:
  serviceRef: { name: grafana, port: 3000 }
  targetGroupARN: arn:aws:elasticloadbalancing:...:targetgroup/grafana-eks-tg/...
  targetType: ip
  networking:
    ingress:
      - from: [{ securityGroup: { groupID: sg-0alb... } }]
        ports: [{ port: 3000, protocol: TCP }]

If you put a header-trusting proxy behind the ALB, verify the JWT. The ALB passes identity in x-amzn-oidc-data, a JWT signed with ES256 [11]. One independent review found a sidecar that base64-decoded the token without checking the signature and regex-matched the email domain. Any pod that could reach the sidecar’s port directly could sign in as any user, including admins, because the cluster enforced no NetworkPolicy. The fix: verify the signature against the regional public key fetched by kid, check alg == ES256, check that signer equals the ALB’s ARN, and check exp. The ALB puts exp in the signed header, not in the payload. Then add NetworkPolicy so only the ALB can reach the pod (defense in depth at every layer, per the Well-Architected Security Pillar [26]). Doing that properly may require security groups for pods, because the ALB and the pods often share subnets, so an ipBlock cannot tell them apart.

2.4 DNS: the apex in Route53, the registrar reduced to NS + DS

Keep the registrar for registration only: NS records, the DS record, transfer lock, hardware 2FA and auto-renew. Put every record in Route53, managed by Terraform.
Delegate child zones (internal.example.dev) when a different writer needs them, for example external-dns with IAM scoped to that zone only.
Mail authentication at the apex: SPF (v=spf1 include:<your-provider> ~all, moving to -all later), DKIM for every sender, and DMARC that ramps p=none → p=quarantine → p=reject behind clean aggregate reports [16]. Cross-domain reporting needs the authorization TXT record at <sending-domain>._report._dmarc.<reporting-domain>, or reporters silently drop your reports (RFC 7489 §7.1).
DNSSEC signed by Route53 with a KMS-backed key-signing key (asymmetric ECC_NIST_P256, in us-east-1), one KSK per zone [5]. Publishing the DS at the registrar is a separate manual step.
CAA at the apex, listing exactly the CAs your hosts actually use [15]. Find them by probing, not by assuming:

for h in $(cat hosts.txt); do
  printf '%s ' "$h"
  echo | openssl s_client -connect "$h:443" -servername "$h" 2>/dev/null \
    | openssl x509 -noout -issuer
done

A typical result is amazon.com (ACM), letsencrypt.org (Traefik/cert-manager and most PaaS vendors) and pki.goog. That last one is a surprise: some website builders issue through Google Trust Services, not Let’s Encrypt.

resource "aws_route53_record" "caa_apex" {
  zone_id = aws_route53_zone.apex.zone_id
  name    = ""
  type    = "CAA"
  ttl     = 3600
  records = [
    "0 issue \"amazon.com\"",
    "0 issue \"letsencrypt.org\"",
    "0 issue \"pki.goog\"",
    "0 iodef \"mailto:security@example.com\"",
  ]
}

One Terraform pitfall: if your zone file is a JSON document fed into a for_each keyed on "<TYPE> <name>", there can be only one record object per type and name. Adding SPF as a second apex TXT is a duplicate-key error at plan time. Every TXT string goes into the same record’s values array.

2.5 The data tier

module "db_core" {
  source                  = "./modules/database"
  instance_class          = "db.r6g.large"
  multi_az                = true
  publicly_accessible     = false          # never parameterize this to true
  deletion_protection     = true
  skip_final_snapshot     = false
  backup_retention_period = 30
  storage_encrypted       = true
  kms_key_id              = aws_kms_key.data.arn
  parameters = {
    "rds.force_ssl"   = "1"
    "max_connections" = null               # know the formula; see §3.4
  }

  create_proxy                       = true
  proxy_require_tls                  = true
  proxy_idle_client_timeout          = 1800  # the AWS default; clients must be shorter
  proxy_max_connections_percent      = 80    # leave headroom for direct users
  proxy_max_idle_connections_percent = 40
  proxy_connection_borrow_timeout    = 10    # fail fast; the default is 120s
}

# Two Valkey roles, never one
module "cache" {               # rebuildable: safe to evict, safe to lose
  source                     = "./modules/redis"
  node_type                  = "cache.t4g.small"
  number_cache_clusters      = 2
  automatic_failover_enabled = true
  multi_az_enabled           = true
  maxmemory_policy           = "allkeys-lru"
  snapshot_retention_limit   = 1
}

module "kv_durable" {          # primary data: never evict, never lose silently
  source                     = "./modules/redis"
  node_type                  = "cache.r7g.large"
  number_cache_clusters      = 2
  automatic_failover_enabled = true
  multi_az_enabled           = true
  maxmemory_policy           = "noeviction"   # an OOM write error beats silent loss
  snapshot_retention_limit   = 7
  transit_encryption_enabled = true
  at_rest_encryption_enabled = true
}

resource "aws_mq_broker" "events" {
  engine_type         = "RabbitMQ"
  deployment_mode     = "CLUSTER_MULTI_AZ"     # 3 nodes across 3 AZs
  publicly_accessible = false
  host_instance_type  = "mq.m5.large"
  user { username = "mq-bootstrap" password = random_password.mq.result }
  lifecycle {
    ignore_changes  = [user]      # the MQ API cannot update RabbitMQ users in place
    prevent_destroy = true        # replacement = new endpoint + lost queues
  }
}
# Terraform owns the topology root via the RabbitMQ provider (management API):
resource "rabbitmq_vhost" "app" { name = "app" }
resource "rabbitmq_user" "ops_admin" { name = "ops-admin" password = var.rotating_pw tags = ["administrator"] }
resource "rabbitmq_exchange" "deposits" {
  name  = "deposits.events"
  vhost = rabbitmq_vhost.app.name
  settings { type = "fanout" durable = true }
}
# Consumers declare their own durable queues and bindings on connect.

2.6 The RPC proxy

For a chain-facing company, a self-hosted RPC proxy such as eRPC [19] in front of two or more node vendors gives you failover, hedging, caching of finalized data, consensus fan-out for writes, and one place to enforce vendor budgets:

rateLimiters:
  store:
    driver: redis                       # without this, every budget is per pod
    redis: { uri: ${RPC_PROXY_REDIS_URL} }
    cacheKeyPrefix: erpc_rl_prod_       # distinct per environment
  budgets:
    - id: vendor-a-global
      rules:
        - method: '*'
          maxCount: 1000
          period: 1s
upstreamDefaults: {}
providers:
  - id: vendor-a
    vendor: vendor-a
    settings: { apiKey: ${VENDOR_A_KEY} }
    overrides:
      "*":
        rateLimitBudget: vendor-a-global
        rateLimitAutoTune: { enabled: false }   # a cost cap must not grow itself
database:
  evmJsonRpcCache:
    connectors: [{ id: cache, driver: redis, redis: { uri: ${RPC_PROXY_REDIS_URL} } }]
    policies:
      - { network: '*', method: 'eth_getTransactionReceipt', finality: finalized, connector: cache, ttl: 10m }
projects:
  - id: main
    networkDefaults:
      failsafe:
        - matchMethod: eth_sendRawTransaction   # MUST precede '*'
          timeout: { duration: 4s }             # inside the caller's 5s submit timeout
          retry:   { maxAttempts: 2, delay: 500ms }
          hedge:   { delay: 1000ms, maxCount: 2 }
        - matchMethod: '*'
          timeout: { duration: 30s }
          retry:   { maxAttempts: 5, delay: 500ms, backoffFactor: 1.3 }
          hedge:   { delay: 1000ms, maxCount: 2 }

Section 4.7 explains why each of those lines exists.

3. Decisions and tradeoffs

3.1 NLB + ingress controller, or ALB + AWS Load Balancer Controller Ingress?

	NLB (L4) + Traefik/NGINX	ALB (L7) per Ingress/IngressGroup
Routing and middleware	In the controller: one config language for rate limits, auth, CORS, body limits	Listener rules (limited), plus whatever the app does
WebSockets, gRPC, long-lived streams	Transparent at L4	Supported, with idle-timeout tuning
Static IPs / Elastic IPs	Yes	No (needs Global Accelerator)
AWS WAF	Cannot attach [14]	Native (REGIONAL web ACL)
Native OIDC/Cognito auth	No	Yes [11]
Cost	One NLB + LCUs; one LB for many hosts	One ALB per IngressGroup + LCUs
Portability	The controller config moves to any cluster	AWS-specific annotations

Recommendation: NLB + an ingress controller for the public API edge at seed and Series A. Internal ALBs with OIDC for human-facing internal tools. CloudFront + ACM for static sites.

The trap: WAFv2 associates with ALB, CloudFront, API Gateway, AppSync, Cognito user pools, App Runner and Verified Access, but never with an NLB [14]. Teams write “WAF: planned” in their architecture doc and only later discover the edge has to change first. Decide early which way you would go: CloudFront in front of the NLB (CLOUDFRONT-scope ACL in us-east-1; also adds edge caching and shield), or swap the public NLB for an ALB (REGIONAL ACL). Both are a migration, not a toggle.

Revisit when: a partner questionnaire requires WAF; you see layer-7 abuse (credential stuffing, scraping) that per-IP rate limits cannot stop; or you need managed bot control.

3.2 Rate limiting at the edge: per replica or shared?

Traefik’s rateLimit middleware keeps its token bucket in each pod’s memory unless you configure a Redis store [21]. With 2 replicas, a client spread across both gets roughly 2× the configured limit. With an HPA, the real limit becomes a range that grows with load, which is exactly when you need it tightest. Two more traps: with no sourceCriterion, the bucket is keyed on the remote address, so every in-cluster caller behind one NAT IP shares a bucket; and behind an NLB with preserve_client_ip off, everyone shares the NLB’s IPs.

Option	Upside	Downside	Stage
Per-pod buckets, document the multiplier	Zero dependencies	Ceiling = config × replicas	Seed, fixed replica count
Redis-backed buckets	True global limit	Redis on the hot path; define fail-open/closed	Series A with HPA on the edge
WAF rate-based rules	Off-cluster, absorbs floods	Needs a WAF-attachable edge; coarse	Scale, abuse-driven

Whichever you choose, write the effective limit (configured × replicas) in the doc next to the configured one. “1000 req/s” in a config file that actually means “2000–6000 req/s, depending on HPA” misleads everyone who reads it.

3.3 Health checks: dedicated ping port, or traffic port?

A dedicated health endpoint (:8082/ping) is the conventional choice. It does not depend on route configuration, it is cheap, and it does not show up in access logs. But a health check on a dedicated port cannot detect a wrong traffic port. If the target group forwards to :8443 on a pod that is not listening there, :8082/ping still returns 200, and the NLB keeps sending traffic into a closed port [8].

Option	Catches	Misses
Dedicated ping port	Process dead, overloaded	Wrong traffic port, broken TLS/listener config
Traffic port, TCP check	Listener missing	App-level failure
Traffic port, HTTP check on a health route	Most things	Couples health to routing; health-route traffic mixed with real traffic

Recommendation: keep the dedicated port, which is what Traefik is designed for, and treat any change to the traffic port as a multi-phase migration (§4.2). Add a synthetic external probe through the public hostname, so that you are also measuring the path customers actually take.

3.4 RDS: Proxy or not, and what a connection budget actually is

RDS Proxy’s value, in priority order: (1) it absorbs Multi-AZ failover, so clients keep their connections to the proxy while it re-points to the new primary; (2) it multiplexes many client connections onto fewer database connections; (3) it lets you use IAM auth and Secrets Manager rotation without changing apps [1].

Where it does not help, and you should go direct:

Session-scoped state. Advisory locks, SET statements, prepared statements, temp tables and long transactions pin the client to a backend connection, which defeats multiplexing [2]. Worse, a lock-holding design that relies on “client dies → backend dies → lock released” breaks behind a proxy, because the proxy separates client death from backend death. One relayer design held one long-lived direct connection per chain lane for a session advisory lock, on purpose, and sent everything else through the proxy.
Analytical jobs. dbt, BI tools and FDW reads are long and pin-heavy. Point them at a read replica directly.
Low-value, low-volume consumers. Proxy pricing is per vCPU of the underlying instance and runs whether or not anything uses it. For a small analytics instance with three consumers, the honest answer was “don’t add it”. A proxy would also have multiplexed a connection leak in a BI tool without fixing it, which is the wrong reflex.

The connection budget. With no max_connections set, RDS Postgres uses LEAST({DBInstanceClassMemory/9531392}, 5000) [20]. That is about 1,800 on a 16 GiB instance and about 900 on 8 GiB. Do the arithmetic per instance, per consumer:

budget(instance) = Σ over direct consumers (max_replicas × pool_size × pools_per_pod)
                 + proxy's MaxConnectionsPercent × max_connections
                 + per-lane / per-chain connections (a different scaling axis!)
                 + migrations, admin, BI tools, superuser_reserved_connections

Lessons from running that sum against a real fleet:

The proxy existed but nearly every consumer bypassed it. Services were wired to the module’s “direct URI” output instead of the proxy-aware default. Check what each service actually connects to, not what the module offers.
Implicit pool sizes. ORMs that size their pool from CPU count (num_cpus × 2 + 1) can read the node’s CPUs, not the pod’s cgroup limit. On a 16-vCPU node, a pod limited to 500m gets a pool of 33. Always set an explicit connection_limit / max.
Per-chain scaling. Connections that scale with the number of configured chains or lanes do not show up in “replicas × pool”.
A flat “connections > 200” alert was 11–44% of real capacity, depending on instance class. That turned out to be safe, but only by luck. Alert on DatabaseConnections / max_connections per instance, and on proxy DatabaseConnectionsBorrowLatency.

Revisit when: any instance’s worst-case sum passes about 60% of max_connections; you add a consumer with session semantics; a BI tool shows up in pg_stat_activity with hundreds of idle connections.

3.5 Valkey/Redis: cache or datastore?

This is the most consequential decision in the data tier, and it is usually made by accident. Developers reach for Redis because it is fast to build on. Infra provisions “a cache”. Nobody writes down which one it is.

Property	Cache	Durable KV
Can every key be rebuilt from another source?	Yes	No
`maxmemory-policy`	`allkeys-lru` / `allkeys-lfu`	`noeviction`
Topology	2 nodes is nice	2+ nodes, Multi-AZ, automatic failover: mandatory
Snapshots	Optional	Daily, ≥7 days, restore-tested
Alerts	Hit rate, memory	Memory ≥ 80% (page), any eviction = SEV1
Sizing	Bounded by TTLs	Grows with users; capacity plan

Why volatile-lru is the worst default for mixed use (ElastiCache’s default parameter group uses volatile-lru [6]): it evicts only keys that have a TTL. If 23 of your 24 key types have no TTL, eviction cannot free memory. The node fills up, the engine OOMs, and ElastiCache replaces the node. With no replica and no snapshot, the replacement starts empty. You get the worst of both worlds: not safe to fill up like a cache, not durable like a database.

Reserve memory. ElastiCache recommends reserving memory (reserved-memory-percent, 25% by default on current parameter groups) for snapshot fork and copy-on-write and for replication buffers [7]. A node at 99.9% of maxmemory is in danger before it reaches 100%.

Revisit when: a “cache” gains a key type with no TTL; a key type’s source of truth is “only here”; memory grows linearly with user count instead of being bounded by TTLs.

3.6 Amazon MQ for RabbitMQ: who owns what

Terraform owns the root of the topology: vhost, users, permissions and exchanges, the stable rendezvous points producers publish to.
Each consumer declares its own durable queue and bindings on connect. A new consumer is then a code deploy, not an infra change, and a consumer that dies does not leave infrastructure drift behind.
Cost of that choice: a broker restore recreates exchanges, but queues come back only when consumers reconnect, so messages published to an exchange with no bound queue are dropped. Your recovery runbook must say “bring consumers up before producers”. Export broker definitions on a schedule and alert when the export goes stale.
Deployment mode: CLUSTER_MULTI_AZ (3 nodes) for prod; SINGLE_INSTANCE for dev. Use quorum queues on the cluster for replicated durability [18].
Credentials: the Amazon MQ API cannot create or update users on RabbitMQ-engine brokers. A Terraform pattern that “rotates” the bootstrap admin password by forcing broker replacement (new endpoint, lost queues, every consumer down) has shipped in at least one real module. Seed the bootstrap user once, ignore_changes = [user], and rotate a RabbitMQ-native admin through the management API instead.
DLQs: decide per queue. “No DLQs anywhere” means failure behaviour is whatever each consumer’s code happens to do.

3.7 PrivateLink versus peering for operator access

If your dev and prod VPCs share a CIDR (it happens more often than people admit, because both took a module default), you can peer the VPN VPC to only one of them. Interface endpoints (PrivateLink) need no route and no unique CIDR. A typical setup is EKS API, Postgres, RabbitMQ and an admin ingress, each behind a VPC endpoint service with a security group admitting the VPN CIDR.

The TLS-hostname trap: a PrivateLink endpoint gives you a vpce-... DNS name. Postgres with sslmode=require connects fine to it. Amazon MQ’s TLS certificate is issued for the broker hostname, so connecting to the vpce- name fails verification. Map the broker hostname to the endpoint IP (/etc/hosts or a private hosted zone) and connect by the broker hostname. Document this in the output description of the endpoint, where people will find it when they ask for a connection string.

Don’t build a public database path into the module. One module offered publicly_accessible = true plus a “public proxy forwarder” (an NLB + socat ASG), and a dev database was internet-facing by construction, protected only by an empty CIDR allow-list. Remove the capability, not just its use.

3.8 RPC proxy: build, buy or go direct?

Option	Upside	Downside
Services call vendors directly	Simple	N vendor keys in N services; no shared budget; no failover
Hosted aggregator	Zero ops	Another vendor in the critical path; opaque limits
Self-hosted proxy (eRPC, etc.)	One budget, failover, hedging, caching, consensus	You now operate a stateful, config-sensitive service on the hot path of every chain call

Self-host once you have two vendors and more than one service calling chains. That is usually by Series A for anyone running relayers or indexers.

4. Failure modes and field lessons

4.1 The cache that held the money

A deposit service kept account routing, session signatures and client webhook configuration in a single-node Valkey (cache.t4g.micro, about 0.5 GiB). The service had originally used Redis as its primary store, and the migration to Postgres was half done. Infra provisioned it as a cache: no replica, no AOF, snapshot_retention_limit = 0, the default volatile-lru, and no memory alarm.

Memory sat at ≥99.88% for hours. Eviction was active (about 1,700 evictions), but it could free only the few TTL-bearing keys. The node OOMed, ElastiCache replaced it, and about 229k keys went to zero. A routine deploy happened about forty minutes earlier and looked like the cause. It was not.

What turned a data-loss event into a customer outage was the readiness probe. Its 500 ms timeout was shorter than the client’s TLS reconnect, so both pods went NotReady and the ingress returned 503 for about 15 minutes, although the pods were running and could process work. The on-call raised the probe timeout and traffic recovered. Recovering the data took the rest of the day: dozens of deposits reconciled by hand, client configurations rebuilt from other systems, and some session signatures that could never be rebuilt.

The architecture document already said: “primary + replica, noeviction, any eviction is SEV1, daily snapshots, quarterly restore drill.” Production had none of it. The document described the target state and was never checked against reality.

Lessons:

Classify every Redis by the cache/durable table in §3.5, and write the classification into the module call.
Alert on memory % and on evictions from day one. The symptom alert (health check failing) paged in 5 minutes. Nothing ever paged on the cause, which had been building for weeks.
A readiness probe that hard-gates on a dependency removes capacity exactly when the dependency wobbles [17]. Probe timeouts must exceed the client’s reconnect time. Consider letting readiness tolerate a dependency blip and alerting on it instead, and shed load deliberately rather than by accident [23][24].
“It rebuilds itself from traffic” is a claim you have to test. Here, account configuration re-registered only when a user reopened the widget. Accounts that received a deposit without doing so were silently dropped, logged as ignored, and marked processed with no retry.
Restore drills should measure the right thing. A weekly snapshot-restore check passed for two clusters and “failed” for the RPC cache, because every key there had a TTL of 10 minutes or less and was discarded as expired when the RDB loaded. DBSIZE == 0 was the correct result. Check rdb_last_load_keys_expired for a volatile keyspace, not DBSIZE.

4.2 Moving the traffic port blackholed the edge

A team moved Traefik’s websecure from :443 to the non-privileged :8443 so the pods could run as non-root. They made one change: entrypoint address and Service targetPort together. The controller re-registered the whole target group at :8443, including pods that had not yet restarted and were still listening only on :443. The health check hit :8082/ping, which answered on every pod, so those pods were marked healthy and blackholed their share of traffic for 2–3 minutes. Neither the rendered manifest diff nor the Terraform plan showed anything wrong, because both describe the end state, not the transition.

The safe sequence, one merge per phase:

Add the listener. Set ports.websecure and --entrypoints.websecure.address to :8443 while websecureTargetPort stays at the old port. Pods restart and gain a listener. The target group does not move. No traffic changes.
Move the target port. Change only websecureTargetPort → 8443. This changes only the Service, so no pods restart. Gate it on no running pod still declaring the old port:
```
kubectl get pods -n ingress -l app=traefik \
  -o jsonpath='{range .items[*]}{.spec.containers[0].ports[*].containerPort}{"\n"}{end}' \
  | grep -wq 443 && echo "BLOCKED: a pod still declares 443" || echo "safe"
```
Use whole-token matching. A substring match on 443 also matches 8443, and an end-of-line anchor misses because the list ends with 8082. Either mistake opens the gate while a pod on the old port is still live.
Add the HTTP→HTTPS redirect on web, now that TLS-decrypted traffic reliably arrives on websecure. Added earlier, the redirect can catch traffic that should have gone to websecure and loop.

Generalize this: any change where the load balancer and the pods must agree on something (port, protocol, proxy-protocol, TLS) needs an expand → switch → contract sequence. Write down the transition state and test that, because the end state will look fine. The Well-Architected Reliability Pillar’s guidance on making all changes through automation and testing them before production [25] applies to the transition, not just to the destination.

4.3 A dedicated ingress controller picked up someone else’s routes

A second Traefik release created for a product host was given --providers.kubernetescrd.namespaces=<shared-namespace>. It loaded the shared controller’s host-less routes (PathPrefix(/chains)) along with its own, so some product-host requests went to the wrong backend until the product routers were given higher priority. The real fix is a label selector or an IngressClass per controller. When several controllers watch one namespace, each one serves every route that has no host.

4.4 Stale pooled connections behind RDS Proxy

A webhook ingester logged insert timed out after 2000ms and readiness postgres probe timed out in clusters, on one pod at a time, while RDS CPU was at 10%, proxy borrow latency was 6 ms and the event loop was flat. Performance Insights showed sessions waiting in Client:ClientRead, which looks like load but means the database is waiting on the app.

The cause: RDS Proxy’s IdleClientTimeout (default 1,800 s [3]) closed client connections that had been idle for 30 minutes. The Node postgres client had max: 25 and no idle_timeout or max_lifetime, so after a quiet period a burst of webhooks landed on sockets the proxy had already discarded, and each query hung until the app’s 2-second timeout. Traffic was low and bursty, about 0.5 req/s, which is the pattern that triggers this.

Fix in the client, not the proxy: idle_timeout well below the proxy’s idle timeout (300 s against 1,800 s), and max_lifetime well below the proxy’s hard 24-hour client-connection lifetime (20 min) [3]. AWS documents both rules. The same applies to any pool behind any middlebox with an idle reaper: NLB (350 s TCP idle), NAT Gateway (350 s) and PgBouncer.

const sql = postgres(url, {
  max: 10,
  idle_timeout: 300,     // < proxy IdleClientTimeout (1800)
  max_lifetime: 60 * 20, // < proxy's fixed 24h client max life
  connect_timeout: 10,
});

4.5 DNSSEC: the outage you cannot see from inside AWS

Rollback ordering is the whole game. If you disable signing (or delete the KMS KSK) while the DS is still published at the parent, the zone becomes bogus. Every validating resolver returns SERVFAIL for every name in the zone, MX included, so inbound mail stops too [5][4]. The safe order, both for rollback and for deliberate decommissioning:

Remove the DS at the registrar.
Wait at least the DS TTL at the parent (often 1–2 days for some TLDs; check it).
Only then disable signing and destroy the key.

Emergency recovery from a bogus zone is step 1 alone. No Terraform change is needed.

Non-obvious traps:

The VPC resolver does not validate DNSSEC. A bogus zone resolves normally from your pods, your in-cluster synthetic checks and your bastion, while customers see total failure. External DNS monitoring has to use validating resolvers (dig +dnssec @1.1.1.1 and check that the AD flag is set).
dig +norecurse against a TLD server for a DS returns empty, which reads as “not published”. Query a recursive resolver (dig DS zone @1.1.1.1) instead.
Use a known-unsigned domain as a negative control when you check AD-flag logic, so you can tell a real result from noise.
The KSK’s KMS key is tier-0. kms:ScheduleKeyDeletion or DisableKey on it takes the zone offline once the DS is published. Add prevent_destroy, and an IAM guardrail that denies key deletion to CI roles.
Registrar access becomes part of incident recovery, not just change management. A registrar lockout combined with a KSK problem is a total outage with no mitigation until access returns. Auto-renew, hardware 2FA and a second recovery path for the registrar account belong in your DR plan. The realistic way to lose the registrar is a lapsed renewal, not a hostile takeover.

4.6 CAA, ACM and hosts on someone else’s platform

ACM follows CNAMEs when it checks CAA, including CNAMEs on parent names [15][22]. If app.example.com (or its parent) is CNAMEd to a PaaS whose target publishes a CAA set without amazon.com, ACM returns CAA_ERROR. That error is terminal: the request will not retry. Fixes: a leaf CAA 0 issue "amazon.com" at the exact name, which must be an A/ALIAS record because CAA cannot coexist with a CNAME, or issue with Let’s Encrypt, which the PaaS already allows.
CAA climbs across delegations. If a delegated child zone has no CAA records, the apex record governs it. Hosts CNAMEd to targets that publish their own CAA never climb to your apex, which is why a tight apex CAA set does not break your docs site or status page.
Moving a host from a PaaS to your own NLB with zero downtime: stand up the NLB with a wildcard cert you already have; attach the real cert through the Service annotation while the host is still dark; check every NLB IP with curl --resolve; then flip DNS. Let’s Encrypt issuance while the host is still on the PaaS makes it truly zero-window. ACM forces a short gap, because the CNAME has to leave the PaaS before ACM can issue.
Moving a host to CloudFront cannot be zero-window in one apply. CloudFront accepts only ACM certs, and Terraform serializes destroy CNAME → issue → validate → deploy alias → create A/AAAA. Each step waits for the previous one, so the whole apply is the outage, about 6 minutes. Near-zero takes three phases: a short CNAME-free window for issuance, restore the CNAME, add the alias while still on the old host, then a Route53-only swap.
The outage is bounded by the SOA negative TTL, not the record TTL. RFC 2308 caches NXDOMAIN for min(SOA MINIMUM, SOA TTL) [13]. With a 900 s negative TTL, any resolver that queries during the gap can hold NXDOMAIN for 15 minutes after the new record exists. In one measured flip, the authoritative gap was about 6 minutes, followed by a long tail as anycast PoPs expired independently. Lower the SOA before a cutover. Lowering the record TTL alone does nothing for the gap.
CloudFront sends no security headers by default. Leaving a PaaS that sent Strict-Transport-Security silently drops HSTS. Diff curl -sSI between the old and new host and restore headers with a response-headers policy.
A “dark” CloudFront distribution on the default *.cloudfront.net certificate accepts TLS 1.0, because minimum_protocol_version can be set only with a custom cert. Give staging distributions a real cert.
Verifying from a laptop: on some VPN or corporate networks, dig @ns-xxx.awsdns-xx is answered by a cache in the path. The tell is the ra flag and a TTL counting down. Verify against three public resolvers by IP, and confirm the apply itself from the Terraform log. On macOS, dig bypasses the system resolver cache, so a successful dig does not mean the browser can resolve the name.

4.7 The RPC proxy’s budgets were not what they looked like

Four separate surprises, all in one component:

The rate-limit budget grew on its own. In eRPC, setting rateLimitBudget on an upstream auto-populates rateLimitAutoTune with enabled: true [19]. The tuner multiplies the budget by 1.05 every healthy minute, up to maxBudget (100,000). That is useful for staying under a vendor’s throttle, but it means a budget meant as a cost cap is not a cap. Set rateLimitAutoTune.enabled: false next to every budget that exists to control spend.
The budget was per pod. With no rateLimiters.store, the limiter uses in-process memory, so every maxCount applies per replica [19]. Under an HPA of 2–3 pods, “1,000/s” was really 2,000–3,000/s, and the vendor’s account-wide plan cap was the actual limit. Configure driver: redis. Setting store.redis.* without driver silently keeps the memory store.
The shared limiter failed open, quietly. Some proxy builds passed a rediss:// (TLS) URI to their Redis client as a raw address. The connection failed, and every budget fell back to no limit, with a warning-level log that production’s error-level logging never printed. Prove the limiter is live: the remote-store duration metric (erpc_rate_limiter_remote_duration_seconds_count) should be climbing, and fail-open counters (erpc_rate_limiter_failopen_total) should stay flat. Alert on both. Fail-open is the right default for a limiter, but it must be loud.
Provider overrides don’t merge. When several overrides keys match a chain (say "*" and evm:42161), the most specific key wins and only its block applies. Every "*" setting is dropped for that chain, including rateLimitBudget, scoreMultipliers and ignoreMethods. Keep per-method scoring on a provider that has no other override keys. Scoring is relative, so demoting vendor B for a method is equivalent to promoting vendor A.

Other proxy lessons:

Write-path timeouts must fit inside the caller’s timeout. A write that inherited the 30 s read timeout and 5 retries turned a 100 ms node rejection (“intrinsic gas too low”) into about 17 s of retries. The relayer saw only its own 5 s timeout, never learned the transaction was permanently invalid, re-broadcast the same bytes hundreds of times, and a stuck nonce blocked one chain’s lane for hours. Fix: a separate eth_sendRawTransaction failsafe entry, ordered first, with a 4 s timeout and 2 attempts. A broadcast rejection is usually a consensus rule that every upstream will apply the same way.
Shared head state. Without a shared-state store, each pod tracks each upstream’s head independently, and eth_blockNumber can go backwards between requests that land on different pods. Indexers notice this.
The proxy silently boots defaults when it finds no config. It searches fixed paths and starts with built-in defaults if the file is missing, which looks exactly like a clean start. Assert on a config-derived metric after deploy.
Config and its env vars ship through different pipelines. If config arrives through Terraform and env vars through GitOps, adding a ${VAR} takes two merges in the right order: the secret and env var first, then the config that reads it. An unset variable expands to "", and the proxy rejects it at load.
Validate configs against the real image locally: docker run -v cfg.yaml:/erpc.yaml:ro <image> with placeholder secrets. If it is still running at the deadline, the config was accepted. A bad config exits with the reason.

4.8 A transient 429 became a permanent failure

A Solana RPC vendor plan hit its usage cap and returned 429 max usage reached. Wallet creation on the deposit path called getAccountInfo, got the 429, threw, and the deposit was marked with an error code in the service’s TERMINAL_ERROR_CODES set. Deposits that had confirmed on-chain were permanently failed and never retried. Detection came from a downstream partner, because the vendor-failure alert had been paused for an unrelated false-positive and nothing alerted on 429 rate or on that error code. Mitigation needed someone with access to the vendor dashboard, and only one engineer had it. Three of the recoveries also hit Redis↔Postgres drift: the account rows existed in Postgres but not in the Redis read path.

Lessons: classify vendor errors (429, 5xx, timeout) as retryable at the call site; alert on vendor 429 rate and on terminal-failure bursts, not just on dollar-value thresholds; give on-call a break-glass path into every vendor console that can raise a quota; have a second vendor for every chain behind the proxy; and when a cache sits in front of a database, give it a read-through fallback to the source of truth.

4.9 Smaller gotchas worth one line each

EC2 volume available ≠ orphaned. A PV can be Bound but unscheduled. Check kubectl get pv for the volume handle before deleting.
Route53 applies each Terraform record as a separate change batch. Right after an apply you may see a partial zone. Re-check after a minute.
The registrar’s transfer lock must be lifted from inside the account, so transferring the domain away is no escape from an account lockout.
Duplicate NLB name prefixes. Controller-generated names truncate (k8s-<ns>-<svc>-) and collide. Identify LBs by the service.k8s.aws/stack tag plus VPC ID, never by name prefix.
ElastiCache snapshot_name is ForceNew in the AWS provider. If your module exposes it for restores, ignore_changes it, or a later edit will destroy and recreate the cluster.

5. Best-practice checklist

Day 1 (seed)

Public edge: NLB with IP targets, TLS 1.3 policy, ACM cert set through the Service annotation, dedicated ping-port health check, deregistration delay < grace period.
Write the effective edge rate limit (config × replicas) in the architecture doc.
Internal tools on an internal ALB + OIDC; any header-trusting proxy verifies the ALB JWT signature and signer.
Route53 owns all records; registrar has transfer lock, hardware 2FA, auto-renew, and at least two people with recovery access.
SPF, DKIM for every sender, DMARC at p=none with rua reporting; CAA derived from probed issuers.
RDS: Multi-AZ, private subnets, publicly_accessible = false not exposed as a module input, deletion protection, 30-day PITR, rds.force_ssl=1, CMK.
Every app DB pool: explicit size, idle_timeout < proxy idle timeout, max_lifetime < 24 h.
Every Redis classified cache or durable in code; durable ones are Multi-AZ, noeviction, with snapshots.
Alerts: Redis memory ≥ 80%, evictions > 0 on durable stores, DatabaseConnections / max_connections ≥ 70%, proxy borrow latency, MQ queue depth and consumer count.
RPC proxy: two vendors per chain, Redis-backed limiter store, rateLimitAutoTune.enabled: false on cost budgets, write-path failsafe ordered first.

First year (Series A)

Choose the WAF path (CloudFront in front, or NLB → ALB) before a questionnaire forces it.
Runbook for multi-phase edge changes (port, protocol, TLS) with transition-state checks.
DMARC ramp to p=reject; SPF -all.
DNSSEC with a KMS KSK per zone, prevent_destroy on keys and zones, an IAM deny on KSK deletion, external validating DNS monitoring, and the rollback order written into the runbook.
Connection-budget sheet per RDS instance, re-derived whenever a consumer or HPA max changes; analytics on a read replica.
Weekly automated restore checks for RDS and every durable Redis, with assertions chosen per keyspace (expired-on-load for volatile caches) and a freshness alert on the check itself.
RabbitMQ: definitions export with a freshness alert; quorum queues; a “consumers before producers” restore order; per-queue DLQ decision recorded.
RPC proxy limiter liveness and fail-open alerts; vendor 429-rate alerts; break-glass access to vendor consoles for on-call.
Synthetic probes through the public hostname from outside AWS.

At scale (Series B+)

Global rate limits (Redis-backed or WAF rate-based rules) instead of per-pod buckets.
Security groups for pods, or subnet separation, so “only the ALB may reach this pod” can be enforced.
RDS Proxy coverage audit: every non-session-scoped consumer goes through the proxy; IAM auth.
Shared-state and limiter Redis for the RPC proxy sized and monitored like a durable store, because the proxy is now a single point of failure for every chain call.
Split the “edge” (DNS, CDN, certs) into its own Terraform root, with its own apply role.
Game days: DNSSEC rollback, broker restore, Redis primary failover, RPC vendor blackout.

6. Maturity roadmap

Stage	Edge	DNS / certs	Postgres	Redis	MQ	RPC
0 — Prototype	One NLB, one Traefik	Records at the registrar	Single-AZ, public for convenience	One node “cache”	None or single instance	Direct vendor calls
1 — Seed	NLB + ping health check; internal ALB + OIDC	Route53 apex, CAA, SPF/DKIM, DMARC `none`	Multi-AZ, private, proxy for OLTP	Cache/durable split; memory alerts	Amazon MQ cluster; TF owns exchanges	Self-hosted proxy, 2 vendors, Redis limiter
2 — Series A	Documented multi-phase changes; synthetic external probes; WAF path chosen	DNSSEC + validating monitors; DMARC `reject`; zero-window cutover playbooks	Connection budget per instance; read replica for analytics; restore drills	Durable stores restore-tested; evictions = SEV1	Definitions export + freshness alerts; quorum queues	Limiter liveness alerts; write-path failsafes; vendor 429 alerts
3 — Scale	WAF (CloudFront or ALB); global rate limits; SG-for-pods	Edge as its own root; registrar DR rehearsed	Proxy everywhere it fits; IAM auth	Capacity-planned durable KV, or migrate to a real DB	DLQ policy per queue; consumer autoscaling	Proxy treated as tier-0 with game days

The step most teams skip is from 1 to 2. Each item in stage 2 is a check that the things built in stage 1 actually do what the architecture document says. Building the components is the easy part. Proving they work as documented is what separates a design that holds up in an incident from one that only exists on paper.

7. References

AWS, Amazon RDS Proxy (User Guide). https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
AWS, Avoiding pinning an RDS Proxy. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-pinning.html
AWS, RDS Proxy connection considerations (IdleClientTimeout, MaxConnectionsPercent, 24-hour client lifetime). https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-connections.html
AWS, Disabling DNSSEC signing (Route53). https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-dnssec-disable.html
AWS, Configuring DNSSEC signing in Amazon Route 53. https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-dnssec.html
AWS, Valkey and Redis OSS parameters (ElastiCache, maxmemory-policy defaults). https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ParameterGroups.Redis.html
AWS, ElastiCache memory management best practices (reserved memory). https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/best-practices-memory.html
AWS, Health checks for Network Load Balancer target groups. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html
AWS, What is a Network Load Balancer? https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html
Kubernetes SIGs, AWS Load Balancer Controller — Service annotations. https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/
AWS, Authenticate users using an Application Load Balancer (OIDC/Cognito, x-amzn-oidc-data JWT verification). https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html
Kubernetes SIGs, TargetGroupBinding. https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/targetgroupbinding/targetgroupbinding/
IETF, RFC 2308, Negative Caching of DNS Queries (DNS NCACHE). https://www.rfc-editor.org/rfc/rfc2308
AWS, AWS WAF Developer Guide (supported resource types). https://docs.aws.amazon.com/waf/latest/developerguide/waf-chapter.html
AWS, (Optional) Configure a CAA record (ACM). https://docs.aws.amazon.com/acm/latest/userguide/setup-caa.html
IETF, RFC 7489, Domain-based Message Authentication, Reporting, and Conformance (DMARC). https://www.rfc-editor.org/rfc/rfc7489
Google, Site Reliability Engineering, ch. 22 “Addressing Cascading Failures”. https://sre.google/sre-book/addressing-cascading-failures/
AWS, Amazon MQ for RabbitMQ best practices. https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/rabbitmq-best-practices.html
eRPC, Rate limiters (auto-tune defaults, memory vs Redis store). https://docs.erpc.cloud/config/rate-limiters — and Providers: https://docs.erpc.cloud/config/projects/providers
AWS, Quotas and constraints for Amazon RDS (maximum database connections formula). https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html
Traefik Labs, RateLimit middleware. https://doc.traefik.io/traefik/middlewares/http/ratelimit/
IETF, RFC 8659, DNS Certification Authority Authorization (CAA) Resource Record. https://www.rfc-editor.org/rfc/rfc8659
Google, Site Reliability Engineering, ch. 21 “Handling Overload”. https://sre.google/sre-book/handling-overload/
AWS Builders’ Library, Using load shedding to avoid overload. https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/
AWS Well-Architected Framework, Reliability Pillar. https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html
AWS Well-Architected Framework, Security Pillar. https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html