EKS Compute, Capacity Planning and Cost for Startups

Abstract. This guide is for senior DevOps, SRE and platform engineers running production Kubernetes on Amazon EKS at a startup. It is written with crypto and web3 infrastructure teams in mind: companies that run relayers, signers, indexers and RPC proxies, depend on metered third-party RPC providers, and cannot afford to have settlement stall because a pod was stuck in Pending. It covers how to pick a node provisioning model (managed node groups with Cluster Autoscaler, or Karpenter), how to run an arm64/Graviton migration so it actually saves money, and how to size HPAs against database connection budgets. It also covers PodDisruptionBudgets that don’t wedge node drains, how to right-size after a scheduling change, and the pitfalls of running a small shared tooling cluster. On the cost side it explains why Savings Plans and Reserved Instances stay out of Terraform. Finally, it shows how to write a “where we break first” capacity analysis that ranks the ceilings that really bite, which are usually not your pods’ CPU. The lessons come from a mature production platform, generalised so they carry over to other teams.

---

1. The problem space

1.1 Forces specific to a startup

A startup’s compute platform runs under constraints that large-company guidance tends to skip:

• Small team, wide blast radius. Two to four people own the clusters, the databases, the CI/CD, the alerting and the bill. Nobody owns capacity full time. In practice capacity work happens in two modes: during an incident, or as a periodic “scaling readiness” push before a launch or a partner integration.
• Spiky, external demand. On-chain activity, a partner’s launch, an airdrop or a market move can multiply load within minutes. You rarely control when it happens.
• The bill gets noticed. At seed and Series A, AWS is often the second-largest line after payroll. For a typical small production platform, EC2 and RDS together make up 40–50% of the monthly bill. Both are reservable, and both get re-shaped often while the platform is young, which makes committing to them risky.
• Partner and auditor pressure. SOC 2 and enterprise partners ask about change control and reproducibility. One consequence: you cannot let the node AMI float to whatever “latest” happens to be at launch time.
• External quotas you don’t operate. RPC providers, webhook providers and indexer APIs carry plan limits, compute-unit caps and prepaid balances. They often fail before anything in your own cluster does.

1.2 Crypto-specific stakes

For a web3 infrastructure company, capacity failures are rarely just slow responses:

• A relayer or solver that can’t schedule a replica misses fills, and the money is gone.
• A signer that loses its single pod during a node drain stops settlement entirely. Signers are often singletons on purpose, because two concurrent nonce managers are worse than one paused one.
• An indexer that falls behind serves stale state to everything downstream.
• An RPC proxy that hits a shared rate-limit budget returns 429s while CPU sits idle on healthy pods. Every dashboard looks green and users see failures.

The rest of this guide treats capacity as a chain of ceilings. It covers the cluster, the database, the message broker and the external providers, and argues that the useful skill is knowing which ceiling you will hit first.

---

2. Reference design

2.1 Topology

The design has four kinds of capacity, each for a different reason:

Capacity	Mechanism	Why it exists
General pool	Karpenter NodePool	Elastic, provisions per pending-pod requirements, about 25 s to a Ready node in practice, consolidates on its own.
Bootstrap group	Managed node group, min = max = 1, no autoscaler	Hosts the autoscaler controllers themselves, so the thing that creates nodes never depends on nodes it created.
Pinned groups	Managed node group, tainted, single AZ	For workloads that need a warm node already in place before their old pod is torn down (zonal EBS PVC + Recreate strategy). Karpenter has no min-size primitive and is reactive, so it cannot guarantee that.
Escape hatch	Managed node group of the old architecture at desired = 0	A reviewed, Terraform-declared path back to amd64. It costs nothing while empty.

2.2 Karpenter NodePool (opinionated defaults)

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    metadata:
      labels:
        workload: general
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: general
      requirements:
        - { key: kubernetes.io/arch,                 operator: In, values: ["arm64"] }
        - { key: kubernetes.io/os,                   operator: In, values: ["linux"] }
        - { key: karpenter.sh/capacity-type,         operator: In, values: ["on-demand"] }  # spot in dev
        - { key: karpenter.k8s.aws/instance-category, operator: In, values: ["m"] }
        - { key: karpenter.k8s.aws/instance-family,  operator: In, values: ["m7g"] }
        - { key: karpenter.k8s.aws/instance-size,    operator: In, values: ["xlarge", "2xlarge"] }
      expireAfter: 720h          # force periodic replacement; no node runs a 6-month-old AMI
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 15m        # match CA's scale-down-unneeded-time during side-by-side migration
    budgets:
      - nodes: "1"               # at most one voluntary disruption at a time on a small fleet
  limits:
    cpu: "80"                    # a ceiling, not a floor — ~10 nodes worst case at 2xlarge

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: general
spec:
  role: eks-node-role                       # or instanceProfile: reuse the existing node role
  amiSelectorTerms:
    - id: ami-0123456789abcdef0             # PINNED in prod; bumped by a reviewed PR
  # - alias: al2023@latest                  # acceptable in dev only
  subnetSelectorTerms:
    - tags: { karpenter.sh/discovery: prod }
  securityGroupSelectorTerms:
    - tags: { karpenter.sh/discovery: prod }
  metadataOptions:
    httpTokens: required
    httpPutResponseHopLimit: 1              # pods can't reach node IMDS credentials
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs: { volumeSize: 50Gi, volumeType: gp3, encrypted: true }

Three of these choices matter more than they look:

1. limits.cpu is mandatory ([16]). A NodePool without one has no ceiling. A runaway HPA or a crash-looping Job with a large request can then provision until it hits the account’s EC2 vCPU quota. Set it to the old node group’s max_size × vCPUs, and alert when NodePool usage passes 80% of it.
2. Pin the AMI in prod. alias: al2023@latest re-resolves at every launch and every expireAfter rotation, with no diff and no review. That is fine for dev. In prod it means a node image change ships with no change record, which is exactly what an auditor will ask about under change-management controls. Get the ID from the EKS-optimised AMI SSM parameter in Terraform and bump it by PR.
3. Interruption handling needs plumbing. Karpenter drains proactively on Spot interruption warnings, rebalance recommendations, scheduled maintenance and instance state changes. It only does that if an SQS queue receives those EventBridge events and the controller is configured to read it. This is easy to leave out while on-demand-only, then forget when you turn on Spot.

2.3 Self-hosting: who schedules the scheduler?

The autoscaler’s own pods must not run on capacity the autoscaler manages. If Karpenter runs only on Karpenter-provisioned nodes and consolidates the node it is running on, a cold restart has nothing to schedule Karpenter onto. A common subtle version: both Cluster Autoscaler and Karpenter nodes carry the same workload=general label, and CA’s nodeSelector targets that label. CA can then end up hosted on capacity it does not control.

# cluster-autoscaler / karpenter controller values
nodeSelector:
  eks.amazonaws.com/nodegroup: bootstrap     # a label only the bootstrap MNG carries
tolerations:
  - { key: dedicated, value: bootstrap, effect: NoSchedule }

Taint the bootstrap group so nothing else lands there, freeze it at min = max = 1, and keep it out of any autoscaler’s scope.

2.4 Managed node groups in Terraform: the desired_size trap

resource "aws_eks_node_group" "pinned" {
  # ...
  scaling_config {
    min_size     = 1
    max_size     = 2
    desired_size = 1   # INITIAL value only
  }
  lifecycle {
    create_before_destroy = true                     # ami_type / instance_types are ForceNew
    ignore_changes        = [scaling_config[0].desired_size]
  }
}

ignore_changes on desired_size is required ([21]). Without it, every plan fights the autoscaler back to the declared number. It has two consequences that catch people out:

• Terraform’s value for desired_size is fiction. The declared 6 in a file can sit beside a live 0, or the reverse.
• Changing desired_size in config does nothing. Terraform drops the ignored attribute from the API call. EKS enforces minSize ≤ desiredSize ≤ maxSize on every UpdateNodegroupConfig. So raising min_size above the live desired size fails at apply time, and lowering max_size below it fails too. If you remove ignore_changes to force it, you can evict nodes serving traffic.

When you really need to set desired size (bootstrapping, freezing a group), reconcile it inside the same module so depends_on runs in the right direction. update-nodegroup-config is asynchronous, so wait for it:

resource "null_resource" "desired_size_reconcile" {
  count    = var.desired_size_override != null ? 1 : 0
  triggers = { desired_size = var.desired_size_override }
  provisioner "local-exec" {
    command = <<-EOT
      aws eks update-nodegroup-config --cluster-name ${var.cluster} \
        --nodegroup-name ${var.nodegroup} \
        --scaling-config desiredSize=${var.desired_size_override}
      aws eks wait nodegroup-active --cluster-name ${var.cluster} --nodegroup-name ${var.nodegroup}
    EOT
  }
}

Before any min_size or max_size change, read the live value:

aws eks describe-nodegroup --cluster-name prod --nodegroup-name pinned \
  --query 'nodegroup.scalingConfig'

---

3. Decisions and tradeoffs

3.1 Cluster Autoscaler + managed node groups vs Karpenter

	Cluster Autoscaler + MNGs	Karpenter
Model	Scales pre-declared ASGs. Picks a group, ASG launches an instance.	Simulates pending pods, calls EC2 Fleet directly with a flexible instance set.
Time to node	Multi-minute (CA scan → ASG → EC2 → join)	About 25 s to Ready observed on arm64 on-demand
Bin-packing	Fixed instance types per group. You pre-guess shapes.	Picks size per batch of pending pods. Consolidation replaces underused nodes.
Floors	min_size is a real primitive	No min-size primitive. Reactive only.
Ceilings	max_size per group, a hard wall that needs a PR	limits.cpu/memory per NodePool, soft but enforced
AZ/PVC awareness	Needs one group per AZ for zonal PVCs	Native: simulates volume topology
Ops surface	Mature, simple, well understood	CRDs, IRSA, SQS interruption queue, controller upgrades
Drift / AMI	MNG version + launch template	EC2NodeClass AMI selection plus drift detection

Recommendation by stage (see also [1], [2], [23]):

• Seed (fewer than ~8 nodes, one cluster). One or two managed node groups with Cluster Autoscaler are fine. Spend the time on requests, limits and PDBs instead. Karpenter’s benefits grow with fleet size and instance diversity.
• Series A (8–40 nodes, frequent shape changes, a migration in progress). Move the elastic pool to Karpenter and keep managed node groups for floors and pins. The hybrid is the normal end state, not a halfway point.
• Scale. Several NodePools (general, memory-heavy, Spot batch), weighted NodePools for Spot-first with on-demand fallback, and disruption budgets with schedules.

Signals to revisit: someone has raised max_size by PR more than twice in a quarter, or pods sit Pending for minutes during bursts. Another signal is keeping per-AZ groups only to satisfy zonal PVCs. A fourth is nodes running well below requested capacity because group instance types don’t fit the workload mix.

How to migrate without an outage: this phased pattern has worked ([20]):

1. Install Karpenter and a NodePool next to Cluster Autoscaler. Prove it with a throwaway pod whose nodeSelector targets karpenter.sh/nodepool=general, a label only Karpenter stamps, so the pod cannot land on CA capacity. Watch a real node launch.
2. Tighten the old group’s max_size to its live desired size (read it from the API first), so new growth goes to Karpenter.
3. Move the autoscaler controllers onto the bootstrap group.
4. Drop the old group’s min_size to 0 and let CA drain it, respecting PDBs. Expect stragglers (see §4.3).
5. Delete the old group once the API confirms desiredSize=0 and zero instances.

Update your alerting before step 4. If a “node not ready” alert excludes CA’s ToBeDeletedByClusterAutoscaler taint, it also needs Karpenter’s karpenter.sh/disrupted taint, or every consolidation pages someone.

3.2 arm64 / Graviton migration

The commonly cited saving is misleading. In us-east-1, on-demand list price for m7g.xlarge is about 15% below m5.xlarge at the same vCPU and memory. That is a real discount, not a transformative one. On one migration we observed, swapping like for like came out roughly cost-flat at the instance level. The old pool ran small, cheap burstable x86 nodes; the new pool used larger general-purpose Graviton nodes, because the smallest burstable arm64 size had an allocatable-memory shortfall after kubelet and system reservations. The clear win was consolidation: a dedicated node group for CPU-heavy workloads disappeared, because those workloads fit on arm64 capacity that existed anyway. Compute spend fell about a third after cutover, almost all of it from fewer nodes rather than cheaper ones.

What to do: model before and after (AWS’s Graviton guide [22] covers the software side; the cost side is yours to model) with real node counts and Pricing API rates, then check Cost Explorer a week after cutover. Don’t put “arm64 is cheaper” in a planning doc as the whole case.

Prerequisites, in order:

1. Multi-arch images for everything that runs on the node. That includes your apps, third-party images, sidecars, init containers, DaemonSets and CronJobs. Build per-arch natively (no QEMU emulation for real builds; it is slow and occasionally wrong) and stitch with a manifest list ([9]). Check upstream base images with docker manifest inspect. Some data tooling images are still amd64-only, and those workloads need an amd64 escape hatch or a rebuilt base.
2. Scan each arch separately. Pulling a manifest list on an amd64 CI runner silently resolves to the amd64 image. If your vulnerability scanner only scans the manifest tag, it never sees CVEs that exist only in the arm64 layer.
3. Audit nodeSelector and affinity in every chart. Comments like “this service is amd64-only” go stale. Check the live pod spec, not the comment.
4. Audit DaemonSet tolerations. See §4.4. This is the one that pages you.

Mechanics: taints don’t move running pods. A common plan is “taint the new arm64 group, migrate services one by one with tolerations, then remove the taint.” Removing a taint changes only future scheduling decisions ([14]). Pods with no explicit nodeSelector stay on the old nodes until something evicts them. That includes the system components nobody manages through CD (CoreDNS, the EBS CSI controller, the load balancer controller, metrics-server, kube-state-metrics). What actually moves them is shrinking the old group’s floor to 0. Even then, Cluster Autoscaler can stall on the last node in a group for 30–60 minutes, with a soft DeletionCandidateOfClusterAutoscaler taint and no PDB blockers. It finishes eventually. To speed it up, cordon the node and restart the deployments on it.

When to revisit: once the fleet is all arm64 and stable, the next lever is instance-family breadth (m6g, m7g, c7g, r7g together) for better bin-packing and Spot availability. Widening your Karpenter requirements gets you more than a further arch change would.

3.3 HPA maxReplicas vs the things HPA can’t see

The HPA formula is desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric) ([12]). Solving for the load multiplier at which maxReplicas stops being enough:

headroom_multiplier = (maxReplicas × targetUtilisation) / (currentReplicas × currentUtilisation)

For a service at 2 replicas, 12% CPU, target 70%, max 3: (3 × 70) / (2 × 12) ≈ 8.8×. That looks comfortable. It hides three things:

1. Utilisation is a percentage of the request. Right-sizing a request from 1000m to 400m multiplies the measured utilisation by 2.5 and divides your headroom by 2.5, with nobody touching maxReplicas. Re-run the headroom maths after every request change.
2. Near-idle CPU tells you little. At 0–3% utilisation you are extrapolating from noise. CPU rarely scales linearly under load: GC pauses, pool contention and downstream latency all bend the curve, and some services hit a non-CPU bottleneck first.
3. The HPA can say “healthy” while the pod can’t schedule. If the node pool is out of room, the HPA raises desired replicas and the new pod sits Pending. On a fleet measured at 83–98% of CPU requested on most nodes, node capacity was the first wall, well before any service’s own maxReplicas.

Policy: set maxReplicas deliberately, ahead of time, from a node-capacity number and a connection-budget number (§3.4), not from a chart default of 3. Raising it during an incident is cheap but slow, and that is the moment you have the least slack.

Chart-ordering gotcha: if your shared Helm library renders the HPA in the same ArgoCD sync wave as the Service, a first install deadlocks. The HPA reports Degraded (FailedGetScale) because its target doesn’t exist yet, and the wave never proceeds to create the Deployment. Render the HPA one wave after the workload. Only fresh installs hit this, which is why it hides behind a fleet of older apps that never did.

3.4 HPA × pool size vs max_connections

Every replica you allow is a set of database connections you have promised. RDS PostgreSQL’s default is max_connections = LEAST(DBInstanceClassMemory / 9531392, 5000) ([6]). That is about 1,800 on a 16 GiB instance and about 900 on 8 GiB. Most teams never set it explicitly, and many have never computed it.

Build the budget table per instance:

Consumer	Replicas (max)	Pool/pod	Path	Worst case
payments-api	HPA 2→20	10	proxy	200
user-api	HPA 2→20	10	proxy	200
indexer	1	~30 (framework default, unconfirmed)	direct	30
scheduler	2	implicit	direct	?
dbt / analytics CronJob	1	long sessions	direct	20–50
Sum vs 1,800

Patterns to look for:

• Implicit pool sizes. Prisma’s default pool is num_physical_cpus × 2 + 1 ([19]). In a container that can read the node’s CPU count rather than the cgroup limit, so moving to a bigger instance type silently enlarges every pool. Always set connection_limit explicitly.
• An RDS Proxy that everything bypasses. A common finding: the proxy exists, but services were wired to the module’s “direct” connection output, so nearly all traffic skips it. Before blaming aggregate growth, check which role holds the connections (pg_stat_activity grouped by usename, application_name).
• Don’t proxy everything. Long-running analytical sessions and SET-heavy transactions pin proxy connections, which defeats multiplexing. Leave them direct. RDS Proxy is billed per vCPU of the instance behind it, every hour it runs. Add it where multiplexing or failover absorption pays for itself ([18]).
• Don’t fix a leak with infrastructure. A BI tool once held over ten times its configured pool size. A proxy would have hidden the leak without fixing it.
• Check your alert threshold against the real number. A flat “connections > 200” alert set without reference to max_connections happened to be conservative everywhere (11–44% of real capacity). That was luck, not design.

3.5 PodDisruptionBudgets

A PDB protects you from voluntary disruption: drains, consolidation, upgrades ([13]). It can also stop a migration dead.

# Library-chart default: never blocks a drain, still limits concurrent eviction
spec:
  maxUnavailable: 1

• minAvailable: 1 on a single-replica workload means zero allowed disruptions. The node can never be drained voluntarily. Cluster Autoscaler won’t evict it. Karpenter won’t consolidate it. One team found nine singleton services configured like this in the middle of a node-group migration. Each needed a manual step: temporarily scale to 2, or delete the pod on an already-cordoned node when the CD system kept reverting the replica count.
• maxUnavailable: 1 as the library default limits concurrent eviction for multi-replica services and still lets a singleton drain. Accept that a singleton has a brief outage on drain. If that outage is unacceptable, the fix is a second replica, not a stricter PDB.
• Real singletons that must not be interrupted mid-operation (a signer mid-batch, a migration Job) should use karpenter.sh/do-not-disrupt: "true" on the pod during the critical window, plus a NodePool disruption budget. Don’t use a PDB that blocks forever.
• Docs drift. An architecture doc claiming “every production workload has a PDB with minAvailable: 1” is either false or a bug report. Keep the claim in the chart, not in prose.

3.6 Right-sizing and requests

• Requests drive scheduling and bin-packing [8]. Limits drive OOMKills and throttling ([15]). A fleet at 95% requested and 40% used means your requests are padded, and that costs real nodes.
• Memory limit = request for anything latency-critical; burstable memory is how one pod’s spike becomes another’s eviction. CPU limits are debatable. Many teams drop them to avoid CFS throttling and rely on requests for fair share.
• Node.js heap vs container limit. A V8 heap target sized from the host’s memory, or a --max-old-space-size above the container limit, produces exit code 137 with reason Error, not OOMKilled, and zero shutdown logs. Set the heap explicitly at about 75% of the limit.
• A step change is not a leak. Memory that was flat for hours and then jumped means an input grew: a config auto-discovered more work, a dataset doubled. Find the input before raising the limit by 50%.

3.7 Savings Plans vs Reserved Instances

Instrument	Covers	Flexibility	Use it for
Compute Savings Plan ([4])	EC2 (including EKS nodes), Fargate, Lambda	Any family, size, AZ, region, OS	EKS compute that autoscales and gets reshaped
EC2 Instance SP / EC2 RIs	One family in one region	Size flexibility within the family only	Rarely right for a startup’s moving fleet
RDS Reserved Instances ([5])	One engine, one family, region; Single-AZ vs Multi-AZ encoded in the offering	Size-flexible within the family; not across families (r6g → r7g) or engines	Steady-state prod databases
ElastiCache reserved nodes	Per node type	Similar to RDS	Only once the spend is worth the lock

Opinionated plan for a young platform (consistent with the Well-Architected cost pillar’s guidance to commit only against stable usage [7], [3]): 1-year, No Upfront, regional. One year because instance families are still moving: a 3-year lock right after an architecture migration is a bet against your own roadmap. No Upfront because cash runway matters more than the extra few points of discount.

Sizing traps:

• Cost Explorer recommendations average over a lookback window. If you just consolidated (compute spend dropped about a third), a 30- or 60-day lookback recommends a commitment sized for the fleet you no longer run. Use the 7-day window, commit below it, and top up later. Savings Plans stack, so starting small and adding a second tranche once 30 days of post-change data exist is better than being over-committed for a year.
• Recommendations lag resizes. If a DB moved classes mid-window, the recommendation under-counts the new class and over-buys the old one. Size RDS RIs 1:1 against live describe-db-instances, not against the average.
• Not everything on the RDS line is reservable. Backup storage, gp3 storage, RDS Proxy and burstable CPU credits aren’t. In one bill only about 75% of RDS spend was instance-hours, so a “40% off” RI saved about 30% of the line. Amazon MQ has no reservation product at all, and Savings Plans don’t apply to Spot.
• Exclude dev databases. They get resized and torn down; a year’s lock on them saves pennies and strands money.
• Check before resizing a reserved DB. A “downsize the read replica from r6g to t4g” ticket strands that replica’s RI for the rest of the term. RIs don’t cross families.

Why none of this is in Terraform. A purchase is a one-way financial commitment. The API has a buy call and no update or delete, and running it again buys a second commitment. Nothing about that fits an idempotent CRUD lifecycle, so it must not be a Terraform resource. What does belong in Terraform is the guardrail:

resource "aws_budgets_budget" "sp_utilisation" {
  name         = "savings-plan-utilisation"
  budget_type  = "SAVINGS_PLANS_UTILIZATION"
  limit_amount = "100"
  limit_unit   = "PERCENTAGE"
  time_unit    = "MONTHLY"
  notification {
    comparison_operator        = "LESS_THAN"
    threshold                  = 95
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = var.cost_alert_emails
  }
}
# …and an identical RI_UTILIZATION budget for RDS.

A stranded commitment has no other symptom. It keeps billing, the instances that are no longer covered quietly go back to on-demand rates, and the total looks unremarkable. Add a spend budget and AWS Cost Anomaly Detection too ([17]). Utilisation budgets alone won’t catch a runaway NodePool.

Document the purchase as a runbook: the exact lookup commands (offering IDs aren’t stable), a check that FixedPrice is 0.0 for No Upfront, the describe-* calls to run first so you don’t buy twice, and verification that utilisation reads about 100% within 24 hours.

3.8 External provider quotas and rate limiters

Your RPC proxy and your webhook provider have ceilings you don’t operate, and they behave differently:

Shape	Example	How it fails	What to measure
Account-wide throughput cap	CU/s on an EVM RPC vendor	429s across all apps on the account	Your share of the account’s peak, not your own traffic alone
Included-quota plan	Webhook credits on a developer tier	Overage billing, possibly an undocumented RPS ceiling	Credits per monitored address × projected users
Prepaid balance	Pay-as-you-go RPC credits	Hard failure at $0	Days of runway at 5× and 10× burn, and alert lead time at those rates
Your own inbound budget	Proxy-level shared rate limiter	429s with idle CPU	Budget ÷ peak inbound

Field lessons:

• Know who else is on the account. One team found its production proxy was only 10–23% of the account’s peak CU/s. The rest came from forgotten test apps. Projecting a 10× increase in prod alone put the account at about 87% of its cap.
• Balance alerts lose lead time as load grows. A $100 warning that gives 1.5 days of notice at today’s burn gives a few hours at 10×. If that vendor is half of your failover pair for 20+ chains, running dry during a load spike leaves you with a single vendor at the worst moment.
• Per-pod vs deployment-wide rate limits. An in-process limiter enforces maxCount per pod. With 2–3 replicas, a configured 1,000 req/s budget really allowed 2,000–3,000. Zero observed rejections “proved” headroom that wasn’t there. Moving the counters to Redis made the documented limit real for the first time, and the measured headroom dropped from about 7.9× to about 3.3×. Two points to take from this. First, if your limiter has a pluggable store, find out which one is active. Second, know which way it fails. A Redis-backed limiter that can’t reach Redis usually fails open (no limit), with a log line at a level prod doesn’t print. Treat a “remote limiter duration” metric that is actually incrementing as proof of life, and alert on fail-open counters.
• Auto-tuning budgets aren’t caps. Some proxies attach an auto-tune block to any budget by default and grow it about 5% per healthy minute. If the budget is a cost control, turn auto-tune off explicitly.
• More proxy replicas don’t raise an inbound budget. The limit is enforced before an upstream is chosen. The fix is one line in config, not an HPA change.
• Don’t trust a gauge just because its name sounds right. A budget_max_count metric reported 100,000 for two budgets configured at 1,000 and 10,000. Cross-check against the config file and against real rejection counters.

---

4. Failure modes and field lessons

4.1 “The HPA isn’t working” (it is; there’s no node)

Symptom: latency climbs, the HPA shows desired 5 / current 3, and someone starts tuning the HPA target. Reality: the new pods are Pending, with 0/7 nodes are available: insufficient cpu. The node pool is at its max_size or NodePool limits, or two nodes are at 93–96% requested. Lesson: in your runbook, put “check for Pending pods and node requested-vs-allocatable” before “tune the HPA”. Alert on kube_pod_status_phase{phase="Pending"} > 0 for 5m in prod namespaces.

4.2 Shrinking the shared tooling cluster starved observability

A team finished its arm64 migration by shrinking the shared tooling cluster, which hosts Grafana, the metrics database, the log store and the CD controller, from four nodes to two. The same day, SEV1 “DatasourceNoData” storms started firing every few hours, each clearing itself in about five minutes, each on a different set of health-check targets. On-call called them false positives because the target services were healthy.

They weren’t false. Both HA replicas of the probe collector, on different nodes in different AZs, logged context deadline exceeded on remote-write at the same moments and dropped samples. The common factor was the single metrics database downstream. It wasn’t erroring; it was too slow to answer inside the exporter deadline. The two remaining nodes were overcommitted: CPU limits at 86% and 206% of capacity, memory limits at 131% and 113%. The collector’s HA design protected against losing one collector, not against the single datastore behind both being briefly starved.

Lessons:

• The tooling cluster is production. Size it for its worst merge or compaction cycle, not its average.
• Limits overcommit is invisible until it bites. kubectl describe node | grep -A8 "Allocated resources" on every node of a small cluster before and after a shrink.
• Install metrics-server or an equivalent everywhere. Without it you cannot confirm throttling after the fact.
• When a NoData alert fires and the target is healthy, check the pipeline: collector exporter logs, metrics-store latency.

4.3 PDBs and stragglers during a node-group drain

Dropping the old group’s floor to 0 drained most nodes. The remaining nodes each held a singleton with minAvailable: 1. Cluster Autoscaler won’t evict those, by design. The fixes were manual: scale to 2 temporarily, or, for workloads whose replica count the CD system kept resetting, cordon and delete the pod directly. Budget a person-afternoon for this in any migration plan, and fix the PDBs first.

4.4 The DaemonSet that finally scheduled everywhere and OOMed

The log-shipper DaemonSet tolerated only control-plane taints. While the arm64 group was tainted dedicated=arm64:NoSchedule, it simply never ran there, and nobody noticed that those nodes’ logs weren’t shipped. A fix added the toleration. Days later the fleet went 100% arm64, and the shipper ran on every node for the first time. Its 512 Mi limit had been sized for its old, partial footprint. Per-node working sets ranged from 67 Mi to 395 Mi depending on what else ran on the node, and the busiest one OOMKilled (SEV2). The fix was 1 Gi limit / 256 Mi request.

Lessons:

• A toleration or nodeSelector fix is a load-shape change, not a harmless scheduling correction. Review resources in the same PR.
• After any taint, label or arch change, run kubectl get ds -A and compare DESIRED with node count. A DaemonSet whose DESIRED is below node count is silently missing coverage.
• If an OOMKill alert fires soon after a scheduling change, suspect “it just started running somewhere new” before “organic growth”.
• DaemonSet memory varies a lot per node. Size for the busiest node, not the mean.

4.5 Terraform said 6, AWS said 0

A capacity doc listed a node group at “desired 6” for months. It was empty. The 6 was a bootstrap value under ignore_changes. A separate change planned to cap a per-AZ group at max_size = 1. A live check showed that AZ running desiredSize = 2 under real load. Applying would either have failed outright or, with ignore_changes removed, evicted a serving node. Live state is the source of truth for anything an autoscaler owns. Write capacity docs from describe-nodegroup and kubectl, and date the snapshot.

4.6 Architecture docs that describe a future that never shipped

One platform’s architecture document said: Cluster Autoscaler provisions nodes; KEDA scales on queue depth; VPA runs in recommendation mode with quarterly reviews; every service has a PDB with minAvailable: 1; load tests run before every major release. In reality: Karpenter had replaced CA, KEDA wasn’t deployed, nobody ran VPA reviews, PDBs defaulted to maxUnavailable: 1, and no load test had ever been run against the capacity ceilings. Every capacity number was a linear extrapolation. Aspirational docs are worse than none, because they answer the 3 a.m. question wrongly and confidently. Mark aspirations as roadmap and describe only what exists now.

4.7 Pinned workloads Karpenter can’t replace

A stateful indexer had a zonal EBS PVC and a Recreate deployment strategy: the old pod must die before the new one attaches the volume. Karpenter can place it correctly, because it simulates volume topology, but it can’t guarantee a warm node already exists in that AZ when the old pod goes away. It is reactive and has no floor. On a 25-second launch that is usually fine. For a component whose restart costs a long resync, it wasn’t. The fix was a small, tainted, single-AZ managed node group with min_size = 1. Don’t force every workload into the elastic pool.

4.8 Stranding a commitment by tidying up

An engineer proposed downsizing an underused read replica (10% average CPU) from a memory-optimised Graviton class to a burstable one, to save money. Had it shipped after the RIs were bought, it would have stranded that replica’s RI for the rest of the term, losing more than the downsize saved. The same applies to a node-group shrink against an instance-locked reservation, which is the reason to prefer Compute Savings Plans for EKS. Add “does this strand a commitment?” to the PR template for any DB resize or node-group change.

4.9 “Where we break first” is rarely CPU

A scaling-readiness exercise ranked nine ceilings by remaining load multiplier. The order:

1. A webhook provider’s included credits: already ~1×, 74% over quota last cycle at current traffic.
2. The general node pool: ~2× to its ceiling, with most nodes 83–98% requested.
3. The RPC proxy’s shared inbound budget: ~3.3×, once it was enforced deployment-wide.
4. An RPC vendor’s account-wide CU/s: ~3.5× nominal, about 87% utilisation projected at 10×.
5. A prepaid RPC balance: a runway ceiling, 17 days of runway → 1.8 days at 10×.
6. The tightest service HPA: ~8.8×, and moot because #2 comes first.
7. Message-broker memory watermark: unmeasured, and the highest blast radius. It blocks publishers cluster-wide, and draining is the only fast recovery.
8. DB connections: within budget, driven mostly by consumers bypassing the proxy.
9. The proxies’ own pod CPU and memory: 50× or more.

The two lowest ceilings were a vendor plan and node bin-packing, and the most dangerous one had never been measured. Pod CPU, the thing most dashboards show, came last.

---

5. How to write a “where we break first” analysis

A one-week exercise any team can run before a launch or a fundraise-driven traffic push:

1. List every ceiling, including cluster (NodePool limits, max_size, EC2 vCPU service quota), workload (maxReplicas), data (max_connections, broker watermark, Redis memory), edge (ingress, inbound rate budgets), and external (vendor caps, balances, credit quotas).
2. For each, compute headroom as ceiling ÷ current peak, with the date of measurement and the source (live API or config). Mark which numbers come from config and which are measured.
3. Rank by headroom, then note blast radius separately. A 7× ceiling that halts all settlement matters more than a 2× ceiling that degrades a dashboard.
4. For each, write the symptom and the lever. “Pending pods → raise NodePool limits.cpu.” “429s with idle CPU → raise inbound maxCount.” “Balance alert → top up; re-threshold for 10× burn.”
5. Turn the ranking into an operator checklist for ambiguous incidents, ordered by likelihood. It costs nothing to rule out a vendor quota first.
6. State the big caveat: without a load test, every multiplier is a linear extrapolation. Real traffic changes non-linearly: GC, pool contention, correlated vendor failure. Schedule the load test (k6 against a prod-shaped environment, ramp to 2× expected peak, hold 10 minutes, pass on error rate, p99, zero OOMKills, no node thrash) and record that it hasn’t happened yet ([10], [11]).
7. Fix the cheap ones now. A one-line budget change or a vendor plan upgrade with lead time shouldn’t wait for an incident.

---

6. Best-practice checklist

Day 1 (seed)

• Every container has explicit CPU and memory requests; memory limit = request for critical services.
• Library-chart PDB default is maxUnavailable: 1; no minAvailable: 1 on singletons.
• HPA rendered one sync wave after its target workload.
• Node groups declare ignore_changes = [scaling_config[0].desired_size], and the team knows what that implies.
• Autoscaler controller runs on capacity it does not manage.
• IMDSv2 required, hop limit 1, on every node template.
• Multi-arch image build in the shared CI template, even if you run one arch today.
• Alerts: Pending pods > 5 min; OOMKilled; node Memory/DiskPressure; DaemonSet desired < nodes.
• Spend budget plus Cost Anomaly Detection on the payer account.
• Every DB connection string sets an explicit pool size.

First year (Series A)

• Karpenter for the elastic pool; managed node groups kept for floors and pins; NodePool limits set and alerted at 80%.
• Prod EC2NodeClass AMI pinned and bumped by PR; expireAfter set.
• arm64 migration modelled with real before/after node counts; per-arch image scanning.
• Per-instance DB connection budget table, recomputed whenever maxReplicas or pool sizes change.
• A “where we break first” document with dated measurements and an operator checklist.
• Vendor quotas: account-wide share known, balance-runway alerts re-thresholded for 10× burn.
• Shared rate limiters backed by a shared store, with proof-of-life and fail-open alerts.
• 1-year No Upfront Compute SP sized from a post-change 7-day window, under the recommendation; RDS RIs sized 1:1 to live prod instances; utilisation budgets in Terraform; purchase runbook.
• Shared tooling cluster headroom checked (requests and limits) before any shrink.

At scale

• Multiple NodePools (general, memory, Spot batch) with weights and disruption budgets on schedules.
• Instance-family breadth for bin-packing and Spot availability.
• Recurring load tests against the ranked ceilings; results stored as a baseline.
• Queue-depth autoscaling (e.g. KEDA) for consumers where CPU is the wrong signal.
• Commitment top-ups on a quarterly cadence; laddered expiries.
• HA for the metrics path (not just Grafana), or a written risk acceptance.

---

7. Maturity roadmap

Stage	Compute	Capacity planning	Cost
Seed	1–2 MNGs + Cluster Autoscaler, one arch	Requests on everything; Pending-pod alert	On-demand; spend budget + anomaly detection
Pre-A / launch prep	Multi-arch images; bootstrap group; PDB hygiene	First “where we break first” doc; DB connection budget	Rightsizing pass; drop idle groups
Series A	Karpenter hybrid; arm64 general pool; pinned MNGs for zonal state	Vendor quota tracking; deployment-wide rate limiters; first load test	1-yr Compute SP + RDS RIs, sized under recommendation; utilisation budgets
Series B+	Multi-NodePool, Spot for batch, disruption schedules	Recurring load tests; capacity reviews tied to launches	Laddered commitments; per-team cost allocation

The order matters. Don’t buy commitments before consolidating, don’t consolidate before PDBs and requests are sane, and don’t trust any capacity number that you haven’t checked against live state.

---

8. References

1. AWS, Amazon EKS Best Practices Guide — Karpenter. https://docs.aws.amazon.com/eks/latest/best-practices/karpenter.html
2. AWS, Amazon EKS Best Practices Guide — Cluster Autoscaler. https://docs.aws.amazon.com/eks/latest/best-practices/cas.html
3. AWS, Amazon EKS Best Practices Guide — Cost Optimization. https://docs.aws.amazon.com/eks/latest/best-practices/cost-opt.html
4. AWS, What are Savings Plans? https://docs.aws.amazon.com/savingsplans/latest/userguide/what-is-savings-plans.html
5. AWS, Reserved DB instances for Amazon RDS. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithReservedDBInstances.html
6. AWS, Quotas and constraints for Amazon RDS (maximum connections). https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html
7. AWS Well-Architected Framework, Cost Optimization Pillar. https://docs.aws.amazon.com/wellarchitected/latest/cost-optimization-pillar/welcome.html
8. AWS Well-Architected Framework, Performance Efficiency Pillar. https://docs.aws.amazon.com/wellarchitected/latest/performance-efficiency-pillar/welcome.html
9. Docker, Multi-platform builds. https://docs.docker.com/build/building/multi-platform/
10. Google, Site Reliability Engineering, ch. 21 “Handling Overload”. https://sre.google/sre-book/handling-overload/
11. Google, The Site Reliability Workbook, ch. 11 “Managing Load”. https://sre.google/workbook/managing-load/
12. Kubernetes, Horizontal Pod Autoscaling. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
13. Kubernetes, Disruptions and Specifying a Disruption Budget. https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ · https://kubernetes.io/docs/tasks/run-application/configure-pdb/
14. Kubernetes, Taints and Tolerations. https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
15. Kubernetes, Resource Management for Pods and Containers. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
16. Karpenter, Disruption (consolidation, budgets, do-not-disrupt) and NodePools. https://karpenter.sh/docs/concepts/disruption/ · https://karpenter.sh/docs/concepts/nodepools/
17. AWS, Getting started with AWS Cost Anomaly Detection. https://docs.aws.amazon.com/cost-management/latest/userguide/getting-started-ad.html
18. AWS, Amazon RDS Proxy. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html
19. Prisma, Connection pool. https://www.prisma.io/docs/orm/prisma-client/setup-and-configuration/databases-connections/connection-pool
20. Karpenter, Migrating from Cluster Autoscaler. https://karpenter.sh/docs/getting-started/migrating-from-cas/
21. HashiCorp, Terraform lifecycle meta-argument and aws_eks_node_group. https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle · https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group
22. AWS, AWS Graviton Getting Started. https://github.com/aws/aws-graviton-getting-started
23. Kubernetes Autoscaler, Cluster Autoscaler FAQ. https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md