Hardened-by-Default Kubernetes Workloads: Library Charts, Pod Security Admission, NetworkPolicy and Policy-as-Code

Abstract. This guide is for senior platform, DevOps and SRE engineers who run application workloads on Amazon EKS at a startup. It is written with crypto infrastructure teams in mind: companies whose pods hold or can use signing keys, relay transactions, call RPC providers and settle on-chain. It describes a layered design. A shared Helm library chart makes the secure configuration the default. Pod Security Admission (PSA) enforces the floor at the API server. NetworkPolicy narrows lateral movement and, eventually, egress. Policy-as-code runs on the rendered manifests in CI, so that per-app overrides cannot quietly wear the defaults away. You will learn how each layer works, what it does not cover, how to roll it out without causing your own outage, and which signals tell you a decision needs revisiting as the company grows. The failure modes are generalized from a real, mature platform. Most of them looked like no-ops in review and turned out not to be.

---

1. The problem: defaults that nobody re-checks

A seed-stage team usually starts with one chart per service, often generated by helm create and edited by hand. By Series A there are thirty such charts. Each was correct on the day it was written, and none have been compared since. Security settings in that world are opt-in. A service is hardened if the engineer who wrote its chart remembered to harden it and nobody relaxed it later to fix a crash at 2 a.m.

The forces at play are specific to startups:

• Small team, wide blast radius. Two or three people own the whole platform, so a control that needs a human to remember it will eventually be forgotten. A control has to be structural: the chart renders it, the API server rejects its absence, or CI fails without it.
• Speed vs safety. Product engineers ship daily. A hardening programme that blocks deploys for a week loses its political capital fast. Rollouts must be staged (audit, warn, then enforce) and ratcheted (no new violations, existing ones tracked down over time).
• Cost. Every admission webhook, service mesh or CNI replacement is something to operate, upgrade and debug. Mechanisms built into the API server (PSA, ValidatingAdmissionPolicy) cost nothing to run and cannot become a webhook outage that blocks every pod create.
• Compliance pressure from partners. Exchanges, custodians and enterprise customers send security questionnaires. These ask about non-root containers, network segmentation, egress control and key isolation by name. NIST SP 800-190 [9] and the NSA/CISA Kubernetes Hardening Guidance [10] are the documents they draw from.
• Crypto-specific stakes. In most SaaS companies, a compromised pod means a data breach. In a crypto company, a compromised pod that can call kms:Sign on a fund-moving key, or read a hot-wallet seed from an environment variable, means irreversible loss. The pods that hold keys are the reason to do this work at all. They are also, as we will see, often the pods with the most exceptions. The CryptoCurrency Security Standard [21] frames key generation, storage and usage as separate control areas. Container hardening and egress control sit squarely in “usage”.

One independent review of a mature platform described the end state of all this in five words: “good defaults, eroded per app.” The platform had an excellent shared library chart, but nothing re-checked the rendered output. When we measured it, that review’s codebase had 69 rendered workloads with 44 policy findings between them: 21 containers with no liveness probe, 8 with no readiness probe, 13 explicitly setting readOnlyRootFilesystem: false, and a floating :latest tag in a deployment gate. 60 of its 61 NetworkPolicies had no egress section at all. Every one of those findings was a deliberate override that someone made for a reason, and none of them had been looked at again.

---

2. Reference design

The design has five layers. Each one catches what the layer before it misses, and each has a different cost and failure behaviour.

2.1 L1 — The library chart

Helm supports type: library charts: they define named templates that application charts include, and they cannot be installed on their own [14]. The pattern that works is a single common library and one thin wrapper chart per app. The wrapper’s templates/ holds a single line per resource kind. All the logic lives in the library.

# charts/payments-api/templates/all.yaml — the entire app chart
{{ include "common.serviceaccount" . }}
---
{{ include "common.deployment" . }}
---
{{ include "common.service" . }}
---
{{ include "common.pdb" . }}
---
{{ include "common.hpa" . }}
---
{{ include "common.networkpolicy" . }}

Security context as a merged baseline, not a values default. The most important design choice is where the defaults live. If they live in each chart’s values.yaml, every copy can drift and a missing key means insecure. Put them in the template as a baseline dict and merge the app’s overrides on top, field by field:

{{- /* pod level */}}
{{- $podBaseline := dict "runAsNonRoot" true "runAsUser" 10001
      "seccompProfile" (dict "type" "RuntimeDefault") }}
securityContext:
  {{- mergeOverwrite $podBaseline (.Values.podSecurityContext | default dict)
      | toYaml | nindent 8 }}

{{- /* container level */}}
{{- $ctrBaseline := dict "allowPrivilegeEscalation" false
      "readOnlyRootFilesystem" true
      "capabilities" (dict "drop" (list "ALL")) }}
securityContext:
  {{- mergeOverwrite $ctrBaseline (.Values.securityContext | default dict)
      | toYaml | nindent 12 }}

Three properties make this work. An app that says nothing gets the full baseline. An app that relaxes one field (readOnlyRootFilesystem: false) keeps every other field. And the relaxation is visible as a literal false in a values file, which is exactly what a policy check can find (L2). Write a template unit test confirming that an explicit false really renders as false with your Helm version. Merge semantics around zero values are the kind of thing that changes quietly between template-function library versions.

The rest of the baseline, each item there for a reason:

Default	Why it is in the library and not per app
automountServiceAccountToken: false on the pod	Most app pods never call the Kubernetes API. A mounted token is a credential an attacker can use for discovery and, with sloppy RBAC, escalation. The pod-spec field overrides the ServiceAccount’s own setting [18], so set it on the pod.
Probes rendered from one common.probe helper	One shape (httpGet or tcpSocket, plus timing) across the fleet. Liveness should be dependency-free (is the process accepting connections?). Readiness can check dependencies.
PDB with maxUnavailable: 1 by default	minAvailable: 1 on a single-replica service blocks every node drain forever. maxUnavailable: 1 never blocks a singleton and still protects multi-replica services.
HPA in a later sync wave than the workload	See §4.6: an HPA whose target does not exist yet reports Degraded, and Argo CD will not move past an unhealthy wave [17].
Soft topology spread, only when effective replicas > 1	Spreading a singleton is noise. Hard spread constraints leave pods Pending during an AZ incident, which is when you need them most.
Env DSL: {secret: "item/FIELD"} rendering secretKeyRef; no envFrom	envFrom on a whole secret injects every key, including ones added later for other consumers. An explicit allow-list per variable is reviewable.
Sync-wave ordering: secrets −1, migrations 0, workload 1, HPA 2	Encodes the dependency order once instead of in 30 charts.
Rollout/Deployment switch, with the HPA’s scaleTargetRef following it	One flag changes the workload kind, and nothing else has to be remembered.

A NetworkPolicy DSL with a separate egress switch. The library should render a policy from a short list of allow rules, and it needs an explicit egressEnabled flag rather than gating on the presence of an egress list. Go templates treat an empty list as false: “the empty values are false, 0, any nil pointer or interface value, and any array, slice, map, or string of length zero” [23]. A template that says {{ if .Values.networkPolicy.egress }} can therefore never express “this service may talk to DNS and nothing else”. That is the most valuable egress posture there is, and it is the one you would want for a signer.

policyTypes:
  - Ingress
  {{- if .Values.networkPolicy.egressEnabled }}
  - Egress
  {{- end }}
{{- if .Values.networkPolicy.egressEnabled }}
egress:
  # Universal allows baked into the library, not repeated per app:
  - to:
      - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
        podSelector: {matchLabels: {k8s-app: kube-dns}}
    ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
  - to:
      - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: telemetry}}
        podSelector: {matchLabels: {app.kubernetes.io/name: otel-collector}}
    ports: [{protocol: TCP, port: 4317}]
  {{- range .Values.networkPolicy.egress }}
  - ... # podSelector / namespaceSelector / cidr / allowAll entries
  {{- end }}
{{- end }}

Bake in the allows that every pod needs (DNS, and your telemetry collector if an environment-wide values layer injects OTEL_EXPORTER_OTLP_ENDPOINT into every pod). Otherwise each per-service rollout has to rediscover them, and one of them will forget.

2.2 L2 — Rendered-manifest policy in CI

The library makes the secure path the easy one, but it cannot stop an override. Admission control (L3) catches only what PSA’s fixed profile covers, and only at deploy time. Missing probes, writable root filesystems, floating tags and absent egress policies are not PSA violations. They show up in production as incidents, not as rejected pods.

The fix is to render every app, in every environment, with the same values stack your GitOps controller uses, and pipe the output into a policy engine on every PR. Conftest runs Rego against any structured file and looks for deny, warn and violation rules [11]. The Kyverno CLI does the same for Kyverno policies [12].

package main

workload_kinds := {"Deployment", "StatefulSet", "DaemonSet", "Rollout"}

pod_specs contains input.spec.template.spec if input.kind in (workload_kinds | {"Job"})
pod_specs contains input.spec.jobTemplate.spec.template.spec if input.kind == "CronJob"

all_containers contains c if {
  some spec in pod_specs
  some field in ["containers", "initContainers"]
  some c in spec[field]
}

deny contains msg if {
  input.kind in workload_kinds
  some c in input.spec.template.spec.containers
  some probe in ["readinessProbe", "livenessProbe"]
  not c[probe]
  msg := sprintf("%s/%s container %q has no %s",
                 [input.kind, input.metadata.name, c.name, probe])
}

# Only an explicit false: unset is fine, the library defaults it to true.
deny contains msg if {
  some c in all_containers
  c.securityContext.readOnlyRootFilesystem == false
  msg := sprintf("%s/%s container %q sets readOnlyRootFilesystem: false",
                 [input.kind, input.metadata.name, c.name])
}

warn contains msg if {
  input.kind == "NetworkPolicy"
  not "Egress" in object.get(input.spec, "policyTypes", [])
  msg := sprintf("NetworkPolicy/%s has no Egress policyType", [input.metadata.name])
}

Design notes that matter in practice:

• Ratchet, don’t gate. On day one you will have dozens of findings. Commit them as a baseline (one line per (rendered-file, message)), fail the PR only on findings not in the baseline, and fail it too when a baseline entry has disappeared without the file being updated, so the baseline only shrinks. Keep messages free of volatile content (no image tags, no line numbers) because the message is the ratchet’s identity.
• Re-check the whole fleet when the policy, the baseline or the ratchet script changes. Otherwise a policy edit is only checked against the one app that happened to be in the PR.
• Cover the pods people forget. Init containers, CronJob templates, migration Jobs, and Argo Rollouts AnalysisTemplate jobs all run real images. The one platform we studied found its canary-analysis job template had never received the security baseline at all. It was the main source of non-compliant pods in its namespace.
• Know what you don’t render. Releases applied by Terraform’s helm_release (ingress controllers, collectors, log shippers) often bypass the GitOps render pipeline completely. List them as a known gap or render them too.
• Render the transition, not only the end state. If your GitOps Application’s list of values files is owned by a different pipeline from the files themselves, a single merge can produce an intermediate state that neither the old nor the new render shows. A mature setup renders that transition explicitly (§4.8).

2.3 L3 — Admission: PSA plus in-process CEL

Pod Security Admission is built into the API server. It is configured per namespace with labels, has three levels (privileged, baseline, restricted) and three modes (enforce, audit, warn) [2]:

apiVersion: v1
kind: Namespace
metadata:
  name: apps
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.31   # pin; bump deliberately
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Be precise about what restricted checks, because teams routinely over- and under-estimate it. The Restricted profile adds controls on volume types, privilege escalation, running as non-root (and as a non-root UID), seccomp and capabilities, on top of Baseline’s prohibitions on hostPath, host namespaces, host ports and privileged containers [1]. readOnlyRootFilesystem is not part of any Pod Security Standard [1]. AWS’s own EKS guidance blurs this: it says PSS “can be used to enforce” a read-only root filesystem [3]. If you want that control enforced, it has to come from your library default plus an L2 rule, or a policy engine. This matters for rollout planning (§4.3): getting to PSA restricted does not need the expensive, app-by-app work of finding write paths.

The second thing to understand: enforce applies only to Pod objects; audit and warn also evaluate workload resources [2]. A Deployment that violates restricted is accepted; its ReplicaSet then fails to create pods, and the only evidence is a FailedCreate event and a ReplicaFailure condition [3]. Running all three modes together is what turns that silent failure into a warning at kubectl apply time and an annotation in the audit log [3].

ValidatingAdmissionPolicy (VAP) handles the cases PSA cannot, such as image provenance, in CEL that runs inside the API server [8]. There is no webhook, so there is no webhook outage that blocks every pod create, and no controller to upgrade. Two mechanics catch people. First, validationActions may not combine Deny with Warn [8]. Go from [Warn, Audit] to [Deny, Audit]. Second, pair the mode with failurePolicy. Use Ignore while auditing, so a broken CEL expression never blocks pods, and Fail once denying, so a broken expression never waves pods through. CEL sees only the object being admitted, so cross-object rules (“every workload has a NetworkPolicy”) belong in CI (L2) or in a policy engine with cluster state.

2.4 L4 — NetworkPolicy: ingress first, then egress

The rollout order is: namespace-wide default-deny ingress; then per-service ingress allow-lists rendered by the library; then egress default-deny, opt-in per service, starting with the services where it matters most.

# Paired with every application namespace; enforce the pairing in CI.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-ingress, namespace: apps}
spec:
  podSelector: {}
  policyTypes: [Ingress]

The cluster must actually enforce it. “Creating a NetworkPolicy resource without a controller that implements it will have no effect” [5]. On EKS, the VPC CNI’s network-policy agent is off by default: you have to set enableNetworkPolicy on the vpc-cni add-on (CNI v1.21.0 or later supports both standard and admin policies) [4][6]. If the CNI was installed self-managed at bootstrap, adopting it as a managed add-on with OVERWRITE conflict resolution will roll the aws-node DaemonSet, which can briefly disrupt pod networking. Schedule it.

resource "aws_eks_addon" "vpc_cni" {
  cluster_name                = aws_eks_cluster.this.name
  addon_name                  = "vpc-cni"
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "OVERWRITE"
  configuration_values        = jsonencode({ enableNetworkPolicy = "true" })
}

The VPC CNI agent’s limits shape the egress design [4]:

• It speaks the upstream NetworkPolicy API: podSelector, namespaceSelector, ipBlock and ports, at L3/L4. It has no FQDN rules. NetworkPolicy itself cannot match DNS names [5].
• Enforcement is optimized for pods that have ownerReferences. Standalone pods (an engineer’s kubectl run debug pod) “might not work reliably” [4].
• A Service’s port must equal its container port.
• There is a limit of 24 port/protocol combinations per CIDR across all policies, and selectors resolve to CIDRs that count toward it.
• Policies apply to the pod’s primary interface only.

Since most crypto workloads call SaaS vendors (RPC providers, bridge and solver APIs, payment processors, block explorers) whose endpoints are CDN-fronted with no stable IPs, per-vendor ipBlock allow-listing is not viable. A workable phased design:

• Phase 1 (VPC CNI only). Default-deny egress for every service that does not call the internet: DNS, the collector, its datastores by subnet CIDR, and its in-cluster callees. Services that call vendors get a coarse 0.0.0.0/0:443 allow, recorded as a residual gap rather than silently accepted. AWS API traffic (KMS, STS, Secrets Manager) moves to interface VPC endpoints, so even key-holding services need no internet egress for signing [22].
• Phase 2 (FQDN egress). Close the :443-anywhere gap with one of the options compared in §3.4: Cilium in CNI-chaining mode with toFQDNs (an in-agent DNS proxy learns the IPs behind allowed names) [7]; AWS Network Firewall with SNI-based domain rules; or an egress proxy/gateway.

# Phase 1 values for a signer that only talks to KMS (via VPC endpoint) and its DB
networkPolicy:
  enabled: true
  ingress:
    - podSelector: {app.kubernetes.io/name: settlement-api}
      ports: [8080]
  egressEnabled: true
  egress:
    - cidr: 10.20.32.0/20      # private subnets: DB + VPC endpoint ENIs
      ports: [5432, 443]

2.5 L5 — Workload identity and the key-holding pod

Every pod that calls AWS should have its own role, through IRSA or EKS Pod Identity. Node instance credentials must be unreachable: use IMDSv2 with a hop limit of 1. IRSA works by exchanging a projected, OIDC-signed service-account token for credentials through STS AssumeRoleWithWebIdentity [19]. Pod Identity instead runs a node agent on the link-local address 169.254.170.23 (ports 80 and 2703, host network). That agent hands out credentials that EKS Auth assumed centrally, and it uses one service principal (pods.eks.amazonaws.com) instead of a per-cluster OIDC provider [20]. Both documents say the same thing plainly: “Containers are not a security boundary,” and per-pod roles do not change that [19][20].

Three interactions with the rest of this design:

1. automountServiceAccountToken: false does not break IRSA or Pod Identity. Both inject their own token volume and environment variables. The automount flag governs only the Kubernetes API token. But see §4.10 for the case where turning it off changed an application’s behaviour anyway.
2. Egress default-deny must leave room for credential delivery. With IRSA, the SDK calls STS, so you need either an STS interface endpoint inside your allowed CIDR or a 443 allow. With Pod Identity, the SDK calls the node-local agent. Test that path explicitly under a default-deny egress policy before rolling it out, and add an explicit ipBlock for 169.254.170.23/32 if your CNI drops it.
3. Network controls are only half of key isolation. The KMS key policy must name the exact role allowed to kms:Sign, and should add aws:SourceVpce so that signing works only through your endpoint [22]. Egress control stops an attacker from exfiltrating through a compromised signer pod. The key policy stops someone with another identity from using the key at all. An admin role that can sign with a fund-moving key from anywhere is a bigger gap than any NetworkPolicy.

---

3. Decisions and tradeoffs

3.1 One library chart vs per-app charts vs an operator

Option	Upside	Downside	Fits
Per-app charts (helm create each)	Zero coupling; each team owns its chart	Defaults drift immediately; fleet-wide changes need 30 PRs	Up to ~5 services
Shared library + thin wrappers	One place for defaults and fixes; overrides are explicit and greppable	A library edit reaches every app in one merge; needs render-diff CI	5–100 services, one platform team
Platform abstraction (CRD/operator, Score, KubeVela)	Strongest encapsulation; app teams never see a pod spec	Another controller to run; debugging goes through an extra layer	Many teams, dedicated platform org

Versioning the library. The cheapest setup references the library by file:// at HEAD, so every app picks up a library change in the same merge. That is efficient, but there is no staging at the library level: a bad edit reaches dev and prod together. Make up for it by rendering every dependent app in every environment on each library PR and showing the diff. Move to a versioned library (publish common to an OCI registry, pin per environment, promote dev → prod) when one of these happens: a library change causes a production incident, or more than one team edits the library.

Values layering. A four-layer stack (chart defaults, then environment-wide, then app-common, then app-per-environment) removes most duplication. But every layer is another place a dependency can hide (§4.9). In one codebase, 43% of the leaf keys present in both the dev and prod files were byte-identical, yet only a handful of apps used the shared layer. Add a values.schema.json to the library early. It turns typos in security-relevant keys (securtyContext) from silent no-ops into render failures.

Revisit when: more than ~10% of charts carry bespoke templates outside the library (CronJobs and hand-written Jobs are the usual ones), or the library passes ~1,500 lines without unit tests.

3.2 PSA vs Kyverno/Gatekeeper vs VAP vs CI-only

Mechanism	Coverage	Failure mode	Ops cost
PSA	Fixed PSS profiles on Pods only	Can’t fail open or closed: it’s in the API server	None
VAP (CEL)	Any single-object rule	failurePolicy decides; no webhook	Low: policies are just API objects
Gatekeeper / Kyverno	Arbitrary rules, cross-object, mutation, background audit [13]	Webhook down + Fail blocks every create; Ignore waves everything through	A controller to upgrade, scale and monitor
CI rendered-manifest policy	Anything you can render	Bypassed by kubectl apply, Terraform-managed releases, or a skipped check	Low; runs in existing pipelines

AWS frames the choice like this: PSS is simpler and built in, while policy-as-code is “considerably more flexible” but means another component to learn and support [3]. A defensible startup stance is PSA + VAP at admission, and conftest (or the Kyverno CLI) in CI, with no admission webhook. That covers the PSS floor and image provenance in-process, and everything else at PR time, where the feedback is cheapest. Adopt Gatekeeper or Kyverno when you need one of three things: mutation (injecting defaults into third-party charts you don’t control), cross-object admission rules, or continuous audit of live resources against rules that PSA doesn’t cover.

Revisit when: an incident involves a resource that was applied outside the GitOps path (a kubectl apply or a Terraform helm_release). CI can’t see those, and admission can.

3.3 How strict to be about readOnlyRootFilesystem

A read-only root filesystem stops an attacker overwriting binaries or dropping tools [3], but it is the setting that breaks applications most often: language runtimes writing caches, temp files, PID files, SQLite. Because it is not PSA-required [1], you can phase it separately:

• Default it to true in the library; mount emptyDir at /tmp and at any known cache path.
• Let apps override it, but only with a comment naming the write path and a ticket, and track every override in the L2 baseline.
• Prioritize removing overrides on key-holding and internet-facing services first. One platform we studied had a recovery-signing service among its four writable-rootfs exceptions in production. That is the wrong place to leave a “temporary” relaxation.

3.4 Egress: VPC CNI vs Cilium vs Network Firewall vs proxy

Option	Granularity	FQDN	Cost/complexity	Notes
VPC CNI NetworkPolicy	Per pod	No	Free; AWS-supported add-on	Phase 1 only; coarse :443 for vendor callers
Cilium (CNI chaining with VPC CNI)	Per pod/identity	Yes (toFQDNs, matchPattern) [7]	A cluster component to run; DNS proxy is in the data path	Best per-service granularity without re-architecting the VPC
AWS Network Firewall (SNI/domain rules)	Per subnet/VPC	Yes (TLS SNI)	Hundreds of USD per month per environment plus data processing; routing changes	AWS-native; can’t tell two pods on the same subnet apart
Egress proxy / gateway (Envoy, Squid, mesh egress gateway)	Per service (via identity or port)	Yes	Bespoke; proxy is a SPOF; apps must honour HTTPS_PROXY	Good audit log of destinations
Security groups for pods	Per pod	No	Branch ENI limits per instance type; pending pods if exhausted [6]	Good for pod-to-RDS; must run standard mode alongside NetworkPolicy [6]

AWS’s own network-security guidance points to Cilium, Calico Enterprise or a mesh egress gateway when you need DNS-hostname rules, and recommends running only one network-policy engine in a cluster [6].

Revisit when: a partner questionnaire asks for egress destination allow-listing per service; when a key-holding service needs to call any internet destination; or when you add a service whose compromise would be existential (a hot-wallet signer). That service should get FQDN-level egress first, even if nothing else does.

3.5 IRSA vs Pod Identity

IRSA needs an IAM OIDC provider per cluster and trust policies that name each cluster’s issuer. Pod Identity uses one service principal and keeps the role-to-service-account mapping in the EKS API, which gives a cleaner separation of duties between IAM admins and cluster admins [20]. Pod Identity is the better default for new clusters on EC2 Linux nodes. It isn’t available on Fargate, Outposts or EKS Anywhere [20]. Migrating an existing fleet buys little security. Do it when you are adding clusters (for example, splitting environments into separate accounts), because that is when per-cluster OIDC trust policies start to multiply.

3.6 Namespaces vs clusters vs accounts for isolation

Kubernetes treats isolation as a spectrum, from namespace-per-tenant through virtual control planes to dedicated clusters [16]. For a startup’s own services, the practical tiers are:

Boundary	Isolates	Doesn’t isolate
Namespace	RBAC, quotas, NetworkPolicy scope, PSA level	Kernel, nodes, cluster-scoped resources, CNI/admission config, the AWS account
Dedicated node group (taints + affinity)	Noisy neighbours; reduces kernel co-tenancy for sensitive pods	API server, cluster RBAC, the account
Cluster	Control plane, admission config, CRDs, cluster-admin blast radius	The AWS account, IAM, KMS
AWS account (under Organizations/SCPs)	IAM, KMS key policies, CloudTrail, quotas	Nothing above it, which is the point

Use namespaces to give workloads different PSA levels. A log-shipping DaemonSet that must mount hostPath cannot live in a restricted namespace, and EKS has no per-workload PSA exemption. Use clusters to separate trust domains with different operators, such as shared internal tooling vs production apps. Use accounts to separate production from everything else. Most of the serious findings in one platform review (deploy roles able to sign with production keys, CI plan roles able to read production secrets) came from a single account in which “admin anywhere = admin everywhere”. No amount of Kubernetes hardening addresses that.

Revisit when: a non-production identity can reach a production key or secret by any path. At that point, per-environment accounts are the fix, not more Deny statements.

---

4. Failure modes and field lessons

These are generalized from a real platform’s incident history and review findings. Most share a shape: the change was correct about the end state and wrong about the transition, or it was a no-op in review and only took effect at the next pod restart.

4.1 NetworkPolicies on a cluster that doesn’t enforce them

A team writes careful ingress allow-lists, reviews them, merges them, and the cluster ignores them because the CNI’s policy agent was never turned on [4][5]. Worse, the existence of the YAML gets cited in security reviews as a compensating control. In one case, an internal authentication proxy trusted a header that it did not cryptographically verify, on the stated assumption that only the load balancer could reach it. The cluster had no policy enforcement, so any pod in the cluster could reach the proxy’s port directly.

Lesson: treat “enforcement is on” as a per-cluster fact to verify, not to assume. Run a canary in each cluster: a pod that tries to reach a denied port, with an alert if it succeeds. AWS recommends exactly this kind of automated violation testing [6]. Never let a NetworkPolicy be the only thing behind an authentication decision.

4.2 Non-root migration: the image must agree with the chart

Flipping runAsNonRoot: true in the chart before the running image tag contains a non-root USER gives CreateContainerConfigError: container has runAsNonRoot and image will run as root, immediately on sync. In a GitOps setup the chart and the image tag are promoted separately. “The Dockerfile PR merged” does not mean “the environment’s pinned tag contains it.”

Lesson: gate every security-context tightening PR on the target environment’s pinned image already containing the matching Dockerfile change. Check the tag, not the source repo.

4.3 os.userInfo() and friends: a numeric UID with no passwd entry

USER 10001 in a Dockerfile satisfies Kubernetes, but some runtimes ask the OS who they are. Node’s os.userInfo() throws ENOENT when the UID has no /etc/passwd entry. A widely used OpenTelemetry SDK calls it during resource detection, so services crashed at startup only after the non-root change. Similar surprises show up in tools that resolve $HOME, in Python’s getpass.getuser(), and in SSH or Git clients.

Lesson: create a real user (useradd -u 10001 -r -s /sbin/nologin app or adduser -D -u 10001 app), not just a numeric USER, and smoke-test startup with tracing and metrics enabled.

4.4 readOnlyRootFilesystem breaks the first write nobody knew about

A first wave of read-only root filesystems caused a live incident, a revert and a re-apply. The cause was write paths the team didn’t know about. The second wave inventoried every write path (temp directories, caches, local files) per service before flipping the setting, and mounted emptyDir volumes for them. It also found that bespoke CronJob and synthetic-check templates outside the library had no security baseline and no volume support. Those had to be fixed in the templates, not in values.

Lesson: separate the PSA-required fields (cheap and mechanical) from readOnlyRootFilesystem (needs knowledge of each app). Ship the first fleet-wide, and the second per service with a write-path inventory.

4.5 PSA dry runs must target a Pod

Teams validate PSA readiness by running kubectl apply --dry-run=server on a Deployment against a namespace labelled enforce: restricted, see no error, and conclude they’re compliant. Enforce never evaluates Deployments [2]. Two techniques actually work:

# 1. Ask the API server what would break, against live pods, without changing anything
kubectl label --dry-run=server --overwrite ns apps \
  pod-security.kubernetes.io/enforce=restricted

# 2. Extract the pod template as a bare Pod and dry-run it into a scratch namespace
kubectl create ns psa-scratch
kubectl label ns psa-scratch pod-security.kubernetes.io/enforce=restricted
kubectl get deploy payments-api -n apps -o json \
  | jq '{apiVersion:"v1",kind:"Pod",metadata:{name:"psa-check",namespace:"psa-scratch"},
         spec:.spec.template.spec}' \
  | kubectl apply --dry-run=server -f -

The error message lists every failing control, not just the first [3].

Lesson: audit/warn for one to two weeks, check the live pod population (including analysis Jobs, migration Jobs and Terraform-managed releases), and only then flip enforce in a separate change. Don’t bundle the label flip into the PR that fixes the last violator. And don’t trust a ticket that says “gated on wave 2 reaching zero overrides” without re-checking: in one rollout, four charts still had a “temporary” root-pin override that had outlived the wave meant to remove it, although the images had been non-root for weeks.

4.6 HPA in the same sync wave as its target

An HPA whose scaleTargetRef doesn’t exist reports Degraded (FailedGetScale). Argo CD holds each wave until the previous one is healthy [17]. Put the HPA in wave 0 alongside the Service and ServiceAccount, and a first install deadlocks: everything lands except the Deployment, which is waiting on a wave that is waiting on it. Existing apps never notice, because their Deployment already exists. The first new service to enable autoscaling finds the bug.

Lesson: encode wave ordering in the library and leave a comment explaining why, so nobody “tidies” it back.

4.7 Migration Jobs that carry the Service selector

A library migration Job that reuses the app’s selector labels becomes a Ready endpoint of the app’s Service while it runs, and receives production traffic on a port nothing is listening on. It’s intermittent, short-lived, and nearly impossible to diagnose from the application side.

Lesson: Jobs, analysis pods and debug pods must have labels that don’t match any Service selector. Add an L2 rule: no non-workload pod template may carry a label set that a Service in the same render selects.

4.8 Moving values between files: the two-merge rule

When the list of values files an Application reads is owned by one pipeline (for example, Terraform) and the files themselves are read straight from Git by the GitOps controller, moving a key from values-prod.yaml into a new shared values-common.yaml in one merge creates a window in which the key is in no file the Application is still reading. One service lost a required environment variable this way and crash-looped for about six minutes. Neither the Terraform plan nor the rendered-manifest diff showed it, because both describe the end state.

Lesson: add the new file and register it (a no-op render), wait for the registration to apply, then delete the duplicated keys. Better still, make CI render the intermediate state.

4.9 Egress allow-lists built from the wrong file

A default-deny egress pilot went through four review rounds. Every round found a real missed dependency:

1. Service URLs defined in the chart’s default values.yaml, not the environment override the author was reading.
2. The empty-list falsy problem, which made a DNS-only service impossible to express (§2.1).
3. The telemetry collector endpoint, injected into every pod by an environment-wide values layer that no single service’s files show.
4. An over-broad security group on the new KMS VPC endpoint (all ports and protocols, when an interface endpoint only ever needs TCP 443).

Lesson: a service’s real dependency list is the union of every values layer plus code defaults. Pilot egress in dev with policy-enforcement logging (the VPC CNI can export policy verdict logs [6]) and watch for drops before going to prod.

4.10 automountServiceAccountToken: false is not always a no-op

Removing the token mount from a fleet of quoting services went cleanly on five of six. On the sixth, which had the heaviest cold start, the new pod never became ready. It retried an outbound WebSocket connection that never succeeded, while the old pods kept serving. A parallel “fix” that tripled the startup-probe threshold only slowed the crash loop down. The leading hypothesis was a library’s credential-provider chain probing for a token file and blocking long enough to starve the connection attempt. Nothing in Kubernetes connects the two.

Lesson: fleet-wide “harmless” changes still go one service at a time for anything with a tight startup budget. Repeated identical errors on every retry are a hard failure to diagnose, not a timing margin to pad.

4.11 Least-privilege RBAC that only bites on restart

An RBAC hardening change removed secrets from an ingress controller’s ClusterRole, reasoning correctly that nothing in its config referenced a Secret. Its Kubernetes provider builds a Secrets informer unconditionally at startup. The running pods kept their existing watches and only logged permission errors, so the change looked like a no-op for hours. A later, unrelated rollout restarted the pods. The provider failed to initialize and served an empty routing table: HTTP 404 on every public host, across every controller sharing the chart. The ticket had specified replacing cluster-wide access with a namespaced Role. Only the removal shipped.

Lesson: a permission or security-context change has not been tested until the pod has restarted under it. Force a rollout in dev as part of the change. Treat a new, persistent permission-denied log from a tier-0 component as an incident, not as noise to suppress.

4.12 Log shippers, hostPath and the namespace that can never be restricted

Node-level log shippers need hostPath mounts for /var/log/pods and the container runtime’s log directory. Baseline (and therefore Restricted) forbids hostPath outright [1], and EKS gives you no per-workload PSA exemption. Your only choices are a separate namespace at privileged (with restricted in audit/warn), or replacing the shipper with something that doesn’t read the host filesystem. Moving a Terraform-managed release to a new namespace is usually ForceNew: uninstall, reinstall, and a fleet-wide restart with a brief log gap.

A related surprise: one team fixed a toleration so that its log DaemonSet finally scheduled onto a newly tainted node group. Shortly afterwards the fleet moved entirely onto that group, and the DaemonSet ran on every node for the first time. Its memory limit had been sized for the old footprint, and it was OOM-killed.

Lesson: record the privileged namespace as a formally accepted standing risk with a named revisit trigger (“when the shipper is replaced”), not a date. After any scheduling change, review resources for the workloads that now run in more places.

4.13 The GitOps project that allows everything

Argo CD’s default project allows any source repo, any destination and any cluster-scoped resource kind [15]. A values PR to any app can therefore render a ClusterRoleBinding into production, and PSA won’t notice because it isn’t a Pod.

Lesson: create one AppProject per trust domain, with sourceRepos, destinations and an explicit clusterResourceWhitelist (often empty for app projects). This is cheap, and it closes the gap that admission control leaves.

4.14 Exceptions outlive their waves

Every hardening wave grants exceptions: a root pin here, a writable rootfs there. These are “until we verify”. Without a mechanism, they become permanent, and they tend to collect on the oddest services. Those are often the bespoke, key-holding ones, which are exactly the services that most need the baseline.

Lesson: each exception goes in the app’s own values file (never the library), carries a one-line reason and ticket, and appears as an entry in the L2 baseline. Review the baseline monthly. A shrinking baseline is the only reliable progress metric.

---

5. Best-practice checklist

Day 1 (seed, or the first EKS cluster)

• One common library chart; every app chart is a thin wrapper. Security context is a merged baseline in the template, not a values default.
• Baseline: runAsNonRoot, fixed non-zero UID, seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false, capabilities.drop: [ALL], readOnlyRootFilesystem: true with emptyDir for /tmp, automountServiceAccountToken: false, CPU/memory requests and limits.
• Liveness (dependency-free) and readiness probes rendered by default; PDB maxUnavailable: 1.
• Dockerfiles create a named user, not just a numeric USER.
• VPC CNI enableNetworkPolicy=true on every cluster, verified with a denied-connection canary.
• Namespace default-deny ingress, plus library-rendered ingress allow-lists.
• PSA audit + warn: restricted on every application namespace from the start.
• IRSA or Pod Identity for every AWS-calling pod; IMDSv2 hop limit 1 on nodes.
• KMS key policies name the exact signing role; no human or CI role can kms:Sign a production key.

First year (Series A)

• Render every app × environment in CI; run conftest/Kyverno CLI on the output, ratcheted against a committed baseline. Rules: probes, no :latest/untagged images, no explicit readOnlyRootFilesystem: false, warn on NetworkPolicy without Egress.
• PSA enforce: restricted per namespace after a clean audit window, flipped in its own PR; pin enforce-version.
• ValidatingAdmissionPolicy for trusted image registries/repositories: [Warn, Audit] with Ignore, then [Deny, Audit] with Fail.
• Interface VPC endpoints for KMS, STS and Secrets Manager; key policies conditioned on aws:SourceVpce.
• Egress default-deny, opt-in per service, starting with key-holding services; coarse :443 allow for vendor callers, documented as residual risk.
• Per-trust-domain Argo CD AppProjects with cluster-resource whitelists.
• values.schema.json on the library; template unit tests for the security baseline.
• Privileged namespaces (log shippers) documented as accepted risks with revisit triggers.

At scale (Series B+)

• Version and promote the library chart (OCI registry, dev → prod pinning).
• FQDN egress (Cilium toFQDNs, a firewall or a proxy) for every internet-calling service, key-holding services first.
• Ratchet the egress warn rule into a deny once the baseline reaches zero.
• Policy engine (Kyverno/Gatekeeper) only if you need mutation of third-party charts, cross-object admission, or continuous audit.
• Signed images verified at admission by digest; production pinned by digest.
• Per-environment AWS accounts; dedicated, tainted node groups or clusters for signers.

---

6. Maturity roadmap

Stage	Library	Admission	Network	Policy-as-code	Identity / keys
0 — Ad hoc	Per-app charts	None	Allow-all	None	Node role or static keys
1 — Defaults	Library with merged baseline	PSA audit/warn	Default-deny ingress; enforcement verified	None	IRSA/Pod Identity; IMDS locked
2 — Enforced floor	+ probes, PDB, HPA waves, schema	PSA enforce restricted; VAP image allow-list	Per-service ingress allow-lists	Rendered-manifest checks, ratcheted	KMS via VPC endpoint; role-scoped key policies
3 — Contained	Versioned, promoted	+ digest/signature verification	Egress default-deny (coarse :443 for vendor callers)	Baseline trending to zero; egress rule warn → deny	Signers on dedicated nodes; no human signing paths
4 — Isolated	Platform abstraction if multi-team	Policy engine where justified	FQDN egress per service	Live-cluster audit plus CI	Per-environment accounts with SCPs

Most startups should aim to reach Stage 2 within a quarter of adopting EKS, and Stage 3 for key-holding services before their first external security audit. Stage 4 should be driven by triggers (a second platform team, a custody partner’s requirements, an incident), not by a calendar.

---

7. Closing

The library chart is the high-leverage piece. A small team can write it in a week, and it fixes the whole fleet at once. On its own, though, it is a starting point that every override wears down. What keeps the defaults intact is a check on the rendered output on every PR, a built-in admission floor that fails closed without a webhook, a network policy that the cluster really enforces, and a key policy that doesn’t trust the network at all. Roll each of these out in two steps: observe first, then enforce. Verify each against live state, not ticket status. Assume every hardening change hasn’t been tested until the pods have restarted under it.

---

References

1. Kubernetes. Pod Security Standards. https://kubernetes.io/docs/concepts/security/pod-security-standards/
2. Kubernetes. Pod Security Admission. https://kubernetes.io/docs/concepts/security/pod-security-admission/
3. Amazon EKS Best Practices Guide. Pod Security. https://docs.aws.amazon.com/eks/latest/best-practices/pod-security.html
4. Amazon EKS User Guide. Limit Pod traffic with Kubernetes network policies. https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html
5. Kubernetes. Network Policies. https://kubernetes.io/docs/concepts/services-networking/network-policies/
6. Amazon EKS Best Practices Guide. Network security. https://docs.aws.amazon.com/eks/latest/best-practices/network-security.html
7. Cilium documentation. Locking Down External Access with DNS-Based Policies. https://docs.cilium.io/en/stable/security/dns/
8. Kubernetes. Validating Admission Policy. https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
9. NIST. SP 800-190: Application Container Security Guide. https://csrc.nist.gov/pubs/sp/800/190/final
10. NSA & CISA. Kubernetes Hardening Guidance, v1.2 (August 2022). https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF
11. Open Policy Agent. Conftest. https://www.conftest.dev/
12. Kyverno. Applying Policies (including in pipelines via the Kyverno CLI). https://kyverno.io/docs/applying-policies/
13. Open Policy Agent. Gatekeeper. https://open-policy-agent.github.io/gatekeeper/website/docs/
14. Helm. Library Charts. https://helm.sh/docs/topics/library_charts/
15. Argo CD. Projects (default project permissions, sourceRepos, destinations, cluster resource whitelists). https://argo-cd.readthedocs.io/en/stable/user-guide/projects/
16. Kubernetes. Multi-tenancy. https://kubernetes.io/docs/concepts/security/multi-tenancy/
17. Argo CD. Sync Phases and Waves. https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/
18. Kubernetes. Configure Service Accounts for Pods (opting out of API credential automounting). https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
19. Amazon EKS User Guide. IAM roles for service accounts. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
20. Amazon EKS User Guide. Learn how EKS Pod Identity grants pods access to AWS services. https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
21. CryptoCurrency Certification Consortium (C4). CryptoCurrency Security Standard (CCSS). https://cryptoconsortium.org/ccss/
22. AWS KMS Developer Guide. Connect to AWS KMS through a VPC endpoint. https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html
23. Go standard library. text/template (definition of empty values in if). https://pkg.go.dev/text/template