Building on AWS · guide 06 of 13

Container Supply-Chain Security on AWS: From Build to Admission

7750 words · about 39 minutes · all guides

Who this is for. Platform, DevOps and security engineers at startups that ship 10–50 containerised services to EKS from GitHub Actions and Amazon ECR. It is written with crypto infrastructure companies in mind: signers, relayers, solvers and settlement services where a forged image becomes stolen funds within minutes.

What you will learn. This guide traces one image from git push to a running pod. At each step it asks who else could have written this? It covers:

multi-arch builds with CodeBuild or buildx;
ECR repository design, including the delete-and-recreate loophole in tag immutability and lifecycle rules that expire images still in production;
CI push-role scoping and the blast radius of a shared role or a shared reusable workflow;
vulnerability scanning and the “gate that exists on paper”;
signing by digest instead of by tag, and which certificate claims you actually have to verify;
SBOMs and SLSA provenance;
digest pinning in GitOps;
admission control with ValidatingAdmissionPolicy, Kyverno or Sigstore policy-controller, rolled out from Audit to Deny.

Every mechanism here was taken from a production platform (the container threats it addresses are catalogued in NIST SP 800-190 [22]), found wanting in a review, and partly fixed. Much of the value is in why the obvious control did less than it seemed to.

1. The problem space

1.1 The question that matters

Most supply-chain programmes start from a tool list: “we have Trivy, we have cosign, we have immutable tags”. A more useful starting point is one question:

What is the full set of identities that could cause a different image to run under the reference production is pinned to?

Call that the image’s write set. For a typical startup it is much larger than the team thinks. It includes:

Every CI identity that can push to the repository. That means every branch, every pull request and every repository whose OIDC subject matches the push role’s trust policy, not just “the release job”.
Every principal that can delete an image or a repository, or flip a repository’s tag mutability. Immutability is a property of the repository object, and it does not survive the repository being deleted.
Every principal that can start the build and choose its build instructions. On AWS CodeBuild, StartBuild takes a --buildspec-override.
Everyone who can change shared build logic. That covers the reusable workflow every service calls at @main, every third-party action, and every tool the build downloads at @latest.
Every upstream you inherit: base images, OS packages, language dependencies.
Everyone who can create pods in the namespace, with any image, from any registry the cluster can reach.
Everyone who can change the admission policy, or the GitOps values file that names the image.

Supply-chain security is the work of shrinking that set, and of making it observable when someone outside it tries to write.

1.2 Why crypto raises the stakes

In a SaaS company, a forged image in a backend service is a data breach. In a crypto company, a forged image in signer-svc is a transfer to an attacker’s address, signed with your key and settled on a chain you cannot roll back. The KMS key policy that limits kms:Sign to the signer’s pod role (see guide 07) does nothing here: the forged image is the signer’s pod, with the signer’s role.

The industry’s worst losses show the pattern. In the 2025 Bybit theft, attackers took AWS session credentials from a wallet vendor’s developer workstation. They then modified the JavaScript bundle in the S3 bucket that served the signing UI, and the injected code changed one targeted transaction [17]. In the 2023 Ledger Connect Kit incident, a phished npm account published a drainer into a library that dApp front ends loaded live, and about $600k was taken in a couple of hours [18]. In the 2025 compromise of the tj-actions/changed-files GitHub Action, a widely used action was retagged to dump CI secrets into public logs [16]. None of these was a cryptographic break. Each was an artifact write path that more people could use than anyone had modelled. In the Bybit case the artifact store itself was the target.

1.3 Forces at a startup

Few engineers, many repositories. One or two platform engineers support 20–40 service repositories. That pushes towards one shared push role, one reusable release workflow and one admission prefix covering “our ECR”. Each of those is a blast-radius multiplier.
Speed. Teams deploy several times a day. Any control that turns an ordinary CI retry into a red build gets disabled within a week. You will see this below with fail_on_critical.
Auditors and partners. SOC 2 (CC6.x on logical access, CC8.1 on change management), exchange and custodian due-diligence questionnaires, and sometimes CCSS ask “are images signed and verified?”. “Signed” is easy to claim. “Verified, against an identity that only main of the right repository can produce” is the claim that holds.
Cost. Amazon Inspector continuous scanning for a few hundred images can cost hundreds of dollars a month. One team estimated $380–980/month for just its internet-facing repositories, against $5–10/month for a scheduled Trivy job doing the same rescan. At seed stage that difference decides the design.

2. Reference design

2.1 The pipeline at a glance

The release path: OIDC into a narrowly scoped role, one build per architecture, a signed manifest list, and the promotion pull request. full size ↗

The design rests on four ideas:

Identity per repository, trusted only from main (or from a protected environment). A pull request in status-page must not be able to mint credentials that can write signer-svc.
The digest is the artifact. Tags are labels for humans. Build, scan, sign, attest, pin and verify all operate on sha256:….
Delete is a write. Immutability without a delete Deny is a speed bump.
Admission verifies what CI asserted. A signature that nothing verifies is a log entry, not a control.

2.2 CI identity: one role per publisher

locals {
  image_publishers = {
    signer-svc   = { repo = "acme/signer-svc",   refs = ["refs/heads/main"] }
    payments-api = { repo = "acme/payments-api", refs = ["refs/heads/main"] }
    # A public repo, or one with a release branch: list refs exactly, never ":*".
    rpc-gateway  = { repo = "acme/rpc-gateway",  refs = ["refs/heads/main", "refs/heads/release"] }
  }
}

data "aws_iam_policy_document" "publish_trust" {
  for_each = local.image_publishers

  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }
    # StringEquals, not StringLike: no wildcard ever reaches this role.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = [for r in each.value.refs : "repo:${each.value.repo}:ref:${r}"]
    }
  }
}

resource "aws_iam_role" "publish" {
  for_each           = local.image_publishers
  name               = "ci-publish-${each.key}"
  assume_role_policy = data.aws_iam_policy_document.publish_trust[each.key].json
}

data "aws_iam_policy_document" "publish" {
  for_each = local.image_publishers

  statement {
    actions   = ["ecr:GetAuthorizationToken"]
    resources = ["*"]
  }
  statement {
    actions = [
      "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:CompleteLayerUpload",
      "ecr:DescribeImages", "ecr:GetDownloadUrlForLayer", "ecr:InitiateLayerUpload",
      "ecr:PutImage", "ecr:UploadLayerPart",
    ]
    resources = [aws_ecr_repository.svc[each.key].arn]   # its own repo, nothing else
  }
  statement {
    actions   = ["codebuild:StartBuild", "codebuild:BatchGetBuilds"]
    resources = values(module.builder[each.key].project_arns)
  }
}

Three details matter here:

Read the subject, don’t infer it. GitHub has changed the default sub format for newer repositories so that it embeds numeric owner and repository IDs (repo:acme@123/svc@456:…) [4]. A trust policy written from the old pattern fails closed. That is safe, but someone will “fix” it with a wider wildcard at 2am. Fetch the real prefix with gh api repos/<org>/<repo>/actions/oidc/customization/sub and record it next to the role.
pull_request subjects match :*. Any workflow that declares id-token: write can request a token, including one triggered by a pull request. For a public repository that means fork contributors, gated only by the “approve first-time contributors” setting [5]. AWS’s own guidance is to restrict sub to specific repositories and branches [5].
Per-repo roles also avoid IAM size limits. One team bolted per-repo StartBuild statements onto a single shared role and hit IAM’s 10,240-byte cap on combined inline policies at the fifteenth repository. Their fix was a wildcard on the naming convention (project/*-amd64). That quietly let every trusted repository start every repository’s build with its own buildspec. A for_each over roles has no such ceiling: the default quota is 1,000 roles per account.

2.3 ECR: immutability, a delete Deny, and age-based lifecycle

resource "aws_ecr_repository" "svc" {
  for_each             = local.image_publishers
  name                 = each.key
  force_delete         = false   # deleting the repo is the ultimate delete+recreate
  image_tag_mutability = "IMMUTABLE_WITH_EXCLUSION"

  # Only genuinely moving tags. Every entry is a tag ANYONE in the write set can repoint.
  dynamic "image_tag_mutability_exclusion_filter" {
    for_each = ["latest", "cache-*"]
    content {
      filter      = image_tag_mutability_exclusion_filter.value
      filter_type = "WILDCARD"
    }
  }

  image_scanning_configuration { scan_on_push = true }
  lifecycle { prevent_destroy = true }
}

data "aws_iam_policy_document" "repo" {
  for_each = local.image_publishers

  statement {
    sid    = "PublisherPushPull"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = [aws_iam_role.publish[each.key].arn]
    }
    actions = [   # deliberately no ecr:BatchDeleteImage
      "ecr:BatchCheckLayerAvailability", "ecr:BatchGetImage", "ecr:CompleteLayerUpload",
      "ecr:GetDownloadUrlForLayer", "ecr:InitiateLayerUpload", "ecr:PutImage", "ecr:UploadLayerPart",
    ]
  }

  statement {
    sid    = "DenyHistoryRewriteExceptBreakGlass"
    effect = "Deny"
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    actions = [
      "ecr:BatchDeleteImage", "ecr:DeleteRepository", "ecr:PutImageTagMutability",
      "ecr:SetRepositoryPolicy", "ecr:DeleteRepositoryPolicy", "ecr:DeleteLifecyclePolicy",
    ]
    condition {
      test     = "ArnNotLike"
      variable = "aws:PrincipalArn"
      values   = [var.break_glass_role_arn, var.registry_terraform_role_arn]
    }
  }
}

The Deny is the part teams leave out. Tag immutability rejects a PutImage that would overwrite a tag [1]. It says nothing about BatchDeleteImage followed by a fresh PutImage under the same now-free tag. And the repository’s resource policy is often where delete gets granted, not the role’s identity policy, so a reviewer who audits only the role misses it. Pair this repository-level Deny with an SCP or permission-boundary Deny on the same actions for every CI role (see guide 03). Test it in a sandbox before relying on it, and confirm that lifecycle expiry, which the ECR service performs itself, still runs.

Lifecycle rules: age-based, with a floor longer than your slowest pin.

{ "rules": [
  { "rulePriority": 1, "description": "untagged > 14d",
    "selection": { "tagStatus": "untagged", "countType": "sinceImagePushed",
                   "countUnit": "days", "countNumber": 14 },
    "action": { "type": "expire" } },
  { "rulePriority": 2, "description": "tagged > 90d",
    "selection": { "tagStatus": "tagged", "tagPatternList": ["*"],
                   "countType": "sinceImagePushed", "countUnit": "days", "countNumber": 90 },
    "action": { "type": "expire" } }
]}

imageCountMoreThan ranks images repository-wide by push time [2]. An image’s survival then depends on how many other images were pushed after it, so a burst of CI builds evicts the tag production is running (§4.6). An age rule makes each image’s fate depend only on its own age. But ECR has no “protect while in use” flag, so the floor has to clear the longest-lived production pin, and a scheduled job has to check that every production pin still exists (§3.3).

2.4 Multi-arch builds that produce a digest, not a tag

With native per-architecture builders (CodeBuild ARM_CONTAINER and LINUX_CONTAINER projects, or GitHub’s arm64 runners), each leg pushes an intermediate image and exports the digest it pushed:

# buildspec-arm64.yml (excerpt)
env:
  exported-variables: [IMAGE_DIGEST]
phases:
  build:
    commands:
      - docker build --platform linux/arm64 -t "$REPO:$SHA-arm64" .
      - docker push "$REPO:$SHA-arm64"
      - IMAGE_DIGEST=$(docker inspect --format '{{index .RepoDigests 0}}' "$REPO:$SHA-arm64" | cut -d@ -f2)

The orchestrating job reads exportedEnvironmentVariables from codebuild batch-get-builds. It then assembles the manifest list from the digests, never from the intermediate tags:

docker buildx imagetools create -t "$REPO:$SHA" \
  "$REPO@$AMD64_DIGEST" "$REPO@$ARM64_DIGEST"
INDEX_DIGEST=$(aws ecr describe-images --repository-name "$NAME" \
  --image-ids imageTag="$SHA" --query 'imageDetails[0].imageDigest' --output text)

Built this way, a mutable -amd64/-arm64 intermediate tag no longer matters. An index references its children by digest, so repointing 3f9c2ab-arm64 afterwards cannot change what 3f9c2ab resolves to. ECR also refuses to expire an image that a manifest list still references [2].

2.5 Scan, sign and attest the digest

for d in "$AMD64_DIGEST" "$ARM64_DIGEST"; do         # per-arch: never scan the index
  trivy image --ignore-unfixed --severity CRITICAL --exit-code 1 \
    --ignorefile .trivyignore.yaml "$REPO@$d"
done

cosign sign   --yes "$REPO@$INDEX_DIGEST"
cosign attest --yes --type spdxjson   --predicate sbom.spdx.json  "$REPO@$INDEX_DIGEST"
cosign attest --yes --type vuln       --predicate trivy.json      "$REPO@$INDEX_DIGEST"
# SLSA provenance: actions/attest (GitHub-generated) or BuildKit --provenance=mode=max

Each choice here is deliberate:

Scan per architecture. Pulling a manifest list on an amd64 runner quietly resolves to the amd64 child, so a critical CVE that exists only in the arm64 layers is never scanned.
Sign only after the gate. If the gate is advisory (§3.6), the signature attests “CI ran”, not “CI approved”. Record the scan result as its own attestation so admission can check it.
Sign @sha256:…, not :tag. Cosign resolves a tag to a digest at signing time. Anything that repoints the tag between push and sign, such as another writer to a mutable tag or a delete-and-recreate, gets signed by your pipeline with your identity.

2.6 Pin by digest in GitOps

# values-prod.yaml, written by the promotion PR
image:
  repository: 111122223333.dkr.ecr.us-east-1.amazonaws.com/signer-svc
  tag: "3f9c2ab@sha256:9b2d…e41"     # sha for humans, digest for the runtime

repo:tag@sha256:… is a valid reference, and the runtime pulls by digest, ignoring the tag. Reviewers keep the readable short SHA. The kubelet gets an immutable reference, and registry-side tag games stop mattering to what runs.

2.7 Admission: two layers

Layer 1: a registry allowlist with ValidatingAdmissionPolicy (VAP). VAP is built into the API server and GA since Kubernetes 1.30 [8]. There is no webhook to run, so no outage can block pod creation, and it suits a coarse “which repositories may run here” rule. A minimal, correct version:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: trusted-image-repositories }
spec:
  failurePolicy: Fail
  matchConstraints:
    namespaceSelector:
      matchExpressions: [{ key: kubernetes.io/metadata.name, operator: NotIn, values: [kube-system] }]
    resourceRules:
      - { apiGroups: [""], apiVersions: [v1], operations: [CREATE, UPDATE], resources: [pods] }
      - { apiGroups: [""], apiVersions: [v1], operations: [UPDATE], resources: [pods/ephemeralcontainers] }
  variables:
    - name: images
      expression: >-
        object.spec.containers.map(c, c.image)
        + (has(object.spec.initContainers) ? object.spec.initContainers.map(c, c.image) : [])
        + (has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers.map(c, c.image) : [])
    - name: qualified   # Docker's rule: a first segment with "." or ":" (or localhost) is a registry
      expression: >-
        variables.images.map(i, i.matches('^([^/]*[.:][^/]*|localhost)/') ? i : 'docker.io/' + i)
    - name: normalized  # "redis" and "docker.io/redis" both mean docker.io/library/redis
      expression: >-
        variables.qualified.map(i, i.matches('^docker[.]io/[^/]+$') ? 'docker.io/library/' + i.substring(10) : i)
    - name: trusted
      expression: >-
        ['111122223333.dkr.ecr.us-east-1.amazonaws.com/signer-svc',
         '111122223333.dkr.ecr.us-east-1.amazonaws.com/payments-api',
         'quay.io/argoproj/']
    - name: untrusted
      expression: >-
        variables.normalized.filter(i, !variables.trusted.exists(t,
          t.endsWith('/') ? i.startsWith(t)
                          : (i == t || i.startsWith(t + ':') || i.startsWith(t + '@'))))
  validations:
    - expression: size(variables.untrusted) == 0
      messageExpression: "'untrusted image repository: ' + variables.untrusted.join(', ')"

Two things in this policy are easy to get wrong:

Normalise before matching. Without it, redis:7 slips past a rule written for docker.io/library/redis.
Anchor the match on : or @ for exact repositories. Without the anchor, …/signer-svc also matches …/signer-svc-debug, and traefik matches traefik-evil.

Layer 2: signature and provenance verification for tier-0 namespaces, with Kyverno verifyImages [10] or Sigstore policy-controller [9]. The certificate identity you check is the subject of §3.7. The short version: if signing happens inside a shared reusable workflow, the certificate’s subject is that shared workflow, and you must also match the calling repository and ref.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-signer-images }
spec:
  webhookTimeoutSeconds: 15
  failurePolicy: Fail
  rules:
    - name: keyless-from-main
      match: { any: [ { resources: { kinds: [Pod], namespaces: [signing] } } ] }
      verifyImages:
        - imageReferences: ["111122223333.dkr.ecr.us-east-1.amazonaws.com/signer-svc*"]
          failureAction: Audit          # → Enforce after a clean audit window
          mutateDigest: true            # rewrite tag → digest at admission
          verifyDigest: true
          required: true
          attestors:
            - entries:
                - keyless:
                    issuer: https://token.actions.githubusercontent.com
                    subject: https://github.com/acme/build-workflows/.github/workflows/release.yaml@refs/heads/main
                    additionalExtensions:
                      githubWorkflowRepository: acme/signer-svc
                      githubWorkflowRef: refs/heads/main
                    rekor: { url: https://rekor.sigstore.dev }

3. Decisions and tradeoffs

3.1 How to build multi-arch images

Option	Upsides	Downsides	Fits
buildx + QEMU on one runner [12]	One job, no infra, no Dockerfile changes	Emulated compile is often 5–20× slower. Rust and C++ builds can time out	Interpreted languages, few services
Cross-compilation in a multi-stage Dockerfile (`TARGETARCH`)	Fast, single runner	Needs toolchain support. Native deps (OpenSSL, `ring`, Prisma engines) get fiddly	Go, and Rust with care
Native per-arch runners (GitHub arm64 runners, or two CodeBuild projects) plus `imagetools create`	Full native speed, no emulation bugs. CodeBuild keeps the build in your AWS account, near ECR, under its own role	Two legs that can fail on their own. You now orchestrate a fan-in	Compiled services, more than ~10 repos
buildx with remote native builder nodes	Native speed, one invocation	Long-lived builder fleet to secure and patch	Larger teams

The tradeoff people miss: partial failure. With two legs, one can succeed and the other die. If your “already built?” guard checks the manifest tag, a retry rebuilds and re-pushes the surviving leg’s tag. One team counted 60 of 1,029 build groups (about 6%) in exactly this half-built state. Making the per-arch tags immutable would have turned every one of those retries into a hard failure. The right fix is not to make intermediates immutable at all costs. It is to make them irrelevant: assemble the index from exported digests (§2.4), and treat the per-arch tags as scratch.

Security of the build itself. CodeBuild running docker build needs privileged_mode = true. Accept that, but give each repository its own CodeBuild service role, scoped to push to its own ECR repository. Treat StartBuild as code execution under that role, because --buildspec-override and --source-location-override let the caller pick both the instructions and the source. Encrypt the source-zip bucket with a CMK and expire objects after a day.

Revisit when build minutes dominate CI cost, or when you need hermetic, reproducible builds for SLSA Build L3 (§3.8).

3.2 Registry layout and tag policy

One repository per deployable is the only layout in which ECR permissions mean anything. A single apps repository with tags like signer-svc-3f9c2ab makes every pusher a writer for every service.

Mutability options [1]:

MUTABLE: never for anything production runs.
IMMUTABLE: the cleanest choice. It breaks any workflow that re-pushes latest on every build. ECR returns ImageTagAlreadyExistsException, and the first time anyone notices is a red release on merge.
IMMUTABLE_WITH_EXCLUSION: immutable except for wildcard-matched tags. This is the pragmatic default. Every exclusion is a tag that anyone in the write set can repoint, so nothing production references may match an exclusion. One team found the API capped exclusion filters at five per repository. A repository with latest plus four channel tags (prod, dev, cache, canary) plus the two per-arch patterns could not be expressed and stayed fully mutable.

Channel tags (prod, stable) are an anti-pattern. A channel tag is mutable by definition, so whatever production resolves through it inherits the widest write set in the repository. Promote by writing a digest into git instead (§3.9).

Pull-through cache for upstream images. If CI pulls from Docker Hub through a shared NAT, you will hit anonymous-pull rate limits (HTTP 429) at the worst moment. An ECR pull-through cache rule fixes that and gives you one place to scan and pin upstreams [13]. Operational sharp edges:

Docker Hub needs credentials in a Secrets Manager secret whose name starts with ecr-pullthroughcache/ [13].
The secret’s keys are username and accessToken, not password. One team found that the wrong key name authenticates as nobody.
It could not be encrypted with a customer-managed KMS key.
Making a cached repository immutable stops ECR refreshing it [13].
Cached repositories get no lifecycle policy or repository policy unless you define a repository creation template [13].

Revisit when you adopt multiple AWS accounts. Put the registry in a dedicated tooling account, with production pulling cross-account through a repository policy that grants pull only. Then even an admin in the production account cannot rewrite history.

3.3 Lifecycle: count, age or pull-based

Rule	Failure mode
`imageCountMoreThan N`	Expires the oldest images repository-wide. Under CI churn, a still-deployed tag falls off the end (§4.6)
`sinceImagePushed N days`	Expires a production pin nobody touched for N days. The slowest-moving service is the victim
`sinceImagePulled`	ECR currently supports it for archive, not expire [2]. Useful as a cost tier, but not a deletion guard

No rule can say “never expire what is running”. The working pattern is therefore:

Use age-based expiry with a floor well above the oldest production pin. One team audited every production values file and found a pin untouched for 58 days, so a 30-day floor would have deleted it on apply. They chose 90.
Run a scheduled pin-existence check: for every production values file, describe-images on the pinned reference, failing loudly when something is missing. Run it with credentials only on the scheduled trigger on the default branch. A pull_request job that can assume a role that reads ECR can also be made to do anything else that role allows.
Remember that nodes cache images. A deleted production image keeps running until a pod lands on a fresh node, so the incident arrives days later, disguised as a capacity event.

Signatures and attestations stored as OCI referrers are expired automatically after their subject image is expired [2]. Under the legacy sha256-<digest>.sig tag scheme, they are just tagged images, subject to your tag rules and your immutability settings.

3.4 Push-role scoping

Option	Write set for `signer-svc`	Complexity
One role, trusted `repo:org/:`	Every workflow, branch and PR in the organisation	Trivial. Also what AWS explicitly warns against [5]
One role, list of repos, each `:*`	Every branch and PR of ~30 repositories	Low. This is the “same as the static key we replaced” trap
One role per repo, `:*`	Every branch and PR of `signer-svc`	Medium
One role per repo, `ref:refs/heads/main` or `environment:release`	Merged, reviewed code in `signer-svc`	Medium. Needs a `role_arn` input in the shared workflow
The above plus registry in a separate account, prod pull-only	Same, and prod admins cannot rewrite either	Higher. Comes with multi-account

The shared role usually exists because it mirrored a static access key it replaced: “no worse than before”. That reasoning holds for blast radius versus the old key. It fails against a pull-request threat model, because OIDC tokens can be minted by pull-request workflows in a way repository secrets never could be. The concrete attack: a pull request to a low-value repository (a status page, a docs site) runs a step that assumes the shared role, then builds and pushes an image into the signer’s repository under a fresh tag. With delete rights it could even use a tag production already pins. A shared role turns your least-reviewed repository into the effective security boundary of your most sensitive one.

Migration hazards are real but bounded. They come in three kinds:

Hardcoded role ARNs. The shared reusable workflow may hardcode the role ARN, so you need a role_arn input and a migration of every caller.
Repos that legitimately push from non-main refs. A dev default branch, v* tags or preview branches each need an explicit entry.
Runbook recovery steps. Some say “re-run the build from a non-default branch”. Change the runbook to “cherry-pick to main”, or define a hotfix environment with required reviewers.

Do the cheap half first: remove delete. It costs a one-line diff, and it closes the “overwrite a prod-pinned tag” path even while the shared role survives.

3.5 Reusable workflows: centralise, then contain the blast radius

A single reusable release workflow is the right call for a startup. It is how you fix a bug once instead of in 22 copy-pasted prepare jobs (one team had to fix a build-context bug in seven repositories one at a time). But it concentrates power:

Calling @main means one push changes build, scan and sign for every service at once. Pin callers to a commit SHA and let Renovate or Dependabot raise the bump PRs, which is GitHub’s own recommendation for third-party workflows [6]. At minimum, require CODEOWNERS review and branch protection on the workflows repository, with the same rigour as production Terraform.
Script injection. ${{ inputs.tags }} interpolated into a run: block is a shell injection sink. GitHub’s guidance is to pass untrusted values through env: [6]. Inputs that look internal, such as tag lists and app names, become attacker-controlled once anyone with write access to any caller repository can call the workflow.
Permissions are a ceiling. A reusable workflow’s job cannot exceed the calling job’s permissions:. Requesting contents: read that the caller didn’t grant fails the whole call at startup with an unhelpful “workflow file issue”. This is a feature: keep callers minimal (id-token: write only, if the build context arrives pre-zipped).
Third-party actions in the workflow are part of every service’s write set. Pin them by full SHA and enforce it with a linter such as pinact or zizmor. The tj-actions compromise [16] hit exactly the teams that referenced a moving tag.
Unchecked tool downloads. curl | sh for kubectl, yq or argo-rollouts@latest in a job holding deploy credentials is the same risk with less visibility. Verify checksums.
Mask the account ID. Logs from a reusable workflow appear under the calling repository’s Actions tab, and that repository may be public.

3.6 Vulnerability scanning: what it can and cannot gate

Where to scan:

Scanner	Coverage	Cost	Notes
ECR basic scan-on-push	OS packages, once	Free	Never re-evaluated after push
Trivy (or Grype) in CI	OS + language packages, at build	Runner minutes	Can gate. Scan per-arch digests
Scheduled Trivy rescan of prod-pinned digests	Same, daily	~$5–10/month egress for a dozen repos	Catches CVEs published after deploy
Amazon Inspector enhanced / continuous [14]	OS + language, re-scans on new CVEs, maps findings to running EKS/ECS workloads	Per image scan and rescan	Doesn’t scan manifest lists themselves [14]. Rescan window defaults to 14 days for newer accounts

fail_on_critical is where good intentions go to die. One platform shipped the gate with default: true, and then 21 of 22 callers set it to false. The documentation still said images “hard-fail on critical findings”. Callers turned it off because almost every critical finding was in layers the team didn’t own:

OS packages in the base image, waiting on a distribution patch;
the bundled npm inside a Node base image, which the app never runs;
a prebuilt Go binary inside a dev-only tool.

None was in application dependencies. A blocking gate the team can’t satisfy gets bypassed, and then the gate is fiction.

A gate that survives contact with reality:

Block on critical and fixable (--ignore-unfixed). Unfixable findings go to a report, not a gate.
Use an allowlist with expiry, not a boolean. Trivy’s YAML ignore file supports per-CVE expired_at and a statement. Every suppression gets an owner and a date, and lapses back into a failure. VEX documents are the standards-based version of the same idea.
Separate deployable images from build-only variants. A test-runner image full of devDependencies shouldn’t block a deploy, but it also shouldn’t be signed with the same identity and predicate as the production image. If you sign what you don’t gate, the signature loses its meaning.
Fix base-image lag structurally. Prefer minimal and distroless bases, rebuild on a schedule (a weekly no-op rebuild picks up OS patches), and pin base images by digest with an automated bump PR. Expect critical findings to appear between rebuilds, and handle them with the 7-day SLA rather than a blocked deploy.
Rescan what’s running. Scan-on-push answers “was it clean when built?”. The daily rescan of production-pinned digests answers “is it clean today?”. Route findings to a ticket, with an alert for internet-facing services.

Revisit when you have more than about 15 internet-facing services or a compliance regime that wants a managed scanner with Security Hub integration. That is when Inspector’s cost buys you something.

3.7 Signing: by digest, keyless, and verified against the right claims

Key-based or keyless. Keyless Sigstore signing (a Fulcio certificate issued for the CI job’s OIDC identity, logged in Rekor) removes key management entirely, and it is the right default for a startup. Key-based signing with an AWS KMS asymmetric key (cosign sign --key awskms:///alias/image-signing) suits teams that cannot depend on the public Sigstore instance, or that must keep signing inside their own audit boundary. AWS Well-Architected recommends signing artifacts after test and verifying them at deploy [19].

By digest. Covered in §2.5. Signing :tag means signing whatever the tag points at when cosign happens to resolve it.

What the certificate actually says. For GitHub Actions, Fulcio embeds several claims, and they are not interchangeable [7]:

Claim (OID)	Value	In a reusable-workflow setup
SAN / Build Signer URI (`.1.9`)	`job_workflow_ref`	The shared workflow, e.g. `…/build-workflows/.github/workflows/release.yaml@refs/heads/main`
Source Repository URI (`.1.12`)	calling repository	`acme/signer-svc`
Source Repository Ref (`.1.14`)	calling ref	`refs/heads/main`, or a PR branch
Build Config URI (`.1.18`)	top-level `workflow_ref`	the caller’s workflow file

The trap: a verification that only matches the subject, e.g. --certificate-identity-regexp '^https://github.com/acme/.+/release.yaml@refs/heads/main$', matches every caller of the shared workflow on every branch. The @refs/heads/main in the subject is the shared workflow’s ref, not the caller’s. A pull request in any repository that calls the workflow produces a signature indistinguishable by subject from a release of signer-svc. Verify the source repository and ref as well:

cosign verify "$REPO@$DIGEST" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity https://github.com/acme/build-workflows/.github/workflows/release.yaml@refs/heads/main \
  --certificate-github-workflow-repository acme/signer-svc \
  --certificate-github-workflow-ref refs/heads/main

Alternatively, sign from the calling workflow, so the subject itself names the service repository. Either way, decide which identity is allowed to produce a production signature per image, and write that down as policy.

Do not “backfill” signatures on the skip path. A common convenience: if the image already exists, skip the build, re-scan and re-sign it “so older images get signatures too”. That signs whatever the tag resolves to today, which may be an image someone else pushed. Re-signing turns an unauthorised push into an authorised-looking one. Sign only digests your own run produced.

3.8 SBOMs and SLSA provenance

SLSA v1.0 defines build levels [3]:

L1: provenance exists.
L2: a hosted build platform generates and signs the provenance.
L3: builds are isolated from each other, and the signing material is inaccessible to user-defined build steps.

Mapping that to the reference design:

L1 comes nearly free: BuildKit --provenance=mode=max --sbom=true, or Syft generating SPDX from the image.
L2 is GitHub artifact attestations (actions/attest with the image digest as subject-digest) [11]. Alternatively, cosign keyless attest from a hosted runner. The platform’s OIDC identity signs, not a key your build script can read.
L3 asks that a malicious step in your build cannot forge provenance. That is hard to claim when the job that runs docker build also holds the signing identity. The usual answer is a trusted builder pattern: provenance generated in a separate job or reusable workflow that the build steps cannot influence (the SLSA GitHub generator model). Most startups should aim for L2 now and L3 for tier-0 services.

Store attestations as OCI referrers next to the image (current cosign uses the OCI 1.1 referrers model [15]). Then they share the image’s lifecycle, and Kyverno or policy-controller can require them at admission, for example “a vuln attestation exists with zero unsuppressed criticals, newer than 7 days”.

SBOMs earn their keep in incident response, not in compliance folders. When the next xz-style backdoor lands, “which running digests contain liblzma 5.6.0?” should be one query over stored SBOMs, not a fleet-wide rescan. NIST SSDF (SP 800-218) [20] and the CNCF supply-chain paper [21] both frame provenance and SBOMs as artifacts to consume, not just produce.

3.9 Pinning production by digest

Pin	Pros	Cons
`:latest` / channel tag	Zero ceremony	Production is whatever was pushed last, by anyone in the write set
`:<short-sha>` on immutable repo	Readable, reviewable	Only as strong as immutability and the delete Deny. 7-char SHAs can collide across repos’ namespaces in tooling
`:<short-sha>@sha256:<digest>`	Immutable at runtime, readable in review	Promotion automation must resolve and write the digest. Diffs are noisier
`@sha256:` only	Purest	Unreadable. Humans stop reviewing

Digest pinning also shrinks admission’s job. With mutateDigest, Kyverno rewrites tags to digests at admission, and policy-controller resolves tags so that the image admitted is the image run [9][10]. That works for workloads you didn’t pin yourself, but pinning in git keeps the source of truth reviewable and lets a drift check compare git with the cluster.

Watch for this: a promotion-exemption rule like “a diff that changes only image.tag needs no extra approval” (see guide 03) must be updated to accept the digest form, and must still reject any other key change in the same file.

3.10 Admission control: which engine, how strict, how to roll out

Engine	Strengths	Weaknesses
ValidatingAdmissionPolicy (built in) [8]	No webhook, so no availability risk. CEL. GA in 1.30	Sees only the admitted object. Cannot verify signatures or fetch registry data
Kyverno [10]	Signature and attestation verification, tag→digest mutation, cert-extension matching, broad policy library	A webhook: its outage plus `failurePolicy: Fail` blocks pod creation. Another controller to upgrade
Sigstore policy-controller [9]	Purpose-built for cosign. Resolves tags to digests. Opt-in per namespace label	Narrower scope. Opt-in means a new namespace is unprotected until labelled
Gatekeeper + Ratify	Fits teams already on OPA	More moving parts

Recommended split: VAP for the cluster-wide registry allowlist, plus Kyverno or policy-controller for signature verification on tier-0 namespaces (signers, relayers, settlement) first. Expand from there.

Designing the allowlist:

Name repositories, not the registry. An entry of <account>.dkr.ecr.<region>.amazonaws.com/ trusts every repository in the account, including test images, mocks and anything any shared-role pusher can create. Combined with a shared push role, that makes the allowlist a no-op against forged images. List production repositories one by one, and keep e2e and test repositories out of production clusters.
Org prefixes (quay.io/argoproj/) trust that upstream organisation completely. Use them only where you would also trust the organisation’s next release.
Docker Hub library/* images are curated, but “any tag” is a lot of trust. docker.io/library/python at any tag is an arbitrary-code runtime for anyone with pod-create rights. Entries like that usually exist for debug and ops pods (kubectl run psql, redis-cli, busybox). Better patterns:
- Mirror the few you need into your own ECR through a pull-through cache with a creation template.
- Pin them by digest in the runbook.
- Keep a dedicated debug namespace with its own narrower policy and RBAC.
Admission does not replace RBAC. It answers “from where”, not “who”. Someone who can create pods can run python -c in an allowed image. Restrict pod-create and pods/exec in tier-0 namespaces separately (guide 05).

Exclusions are holes. Excluding kube-system is standard, because managed add-ons update their own images and a wrong deny there takes down node networking. But anyone who can create pods in kube-system now bypasses the policy, so treat create rights there as cluster-admin.

Failure policy. Use Ignore while in Audit, so that a policy bug can’t block pods, and Fail with Deny, so that a policy that can’t evaluate doesn’t wave pods through. For webhooks, Fail means the webhook’s availability is your deploy availability. Run at least two replicas with a PDB, and alert on webhook latency and errors.

Rollout, Audit → Deny:

Build the allowlist from reality: every image running in the cluster, plus every image the charts and Terraform-managed Helm releases declare.
Bind with ["Audit", "Warn"]. Note that Deny and Warn cannot be combined [8]. Leave it for two or more weeks, including at least one on-call shift of incident debugging, because debug pods are where the surprises come from.
Query audit logs for validation.policy.admission.k8s.io/validation_failure annotations until a full week is clean.
Flip to ["Audit", "Deny"] in its own PR. Write the runbook entry before the flip. After the flip, ArgoCD reports a successful sync while the ReplicaSet emits FailedCreate and the rollout sits in Progressing, which looks like nothing anyone has seen before.
Provide a documented, audited escape hatch: patch the binding back to ["Audit","Warn"] with kubectl. Make sure your IaC reverts the patch on the next apply, so the hatch closes itself.

4. Failure modes and field lessons

These come from one mature platform’s history and an independent review of it, generalised.

4.1 Immutability that didn’t stop overwrite. Tags were immutable, and reviewers checked the push role’s identity policy, which had no delete. The repository policy granted ecr:BatchDeleteImage to the same role on every repository. Any of about 30 trusted repositories’ workflows, on any branch, could delete a production-pinned tag and push a different image under it. The fix was a one-line removal. The lesson: audit the effective permission (identity policy ∪ resource policy − Denies), not the role.

4.2 The shared push role. One OIDC role, trusted :* for every service repository, could push to every ECR repository and start every CodeBuild project. A pull request in an internal analytics dashboard could therefore build and push a settlement-service image. It existed because it faithfully reproduced the static key it replaced. Per-repo, main-only roles were blocked for months on a hardcoded role ARN in the shared workflow. Add the input first, even before you have time to migrate.

4.3 The registry allowlist that trusted the registry. Production admission enforced Deny, but its first entry was “our ECR, every repository”. Combined with 4.2, the allowlist’s only practical effect was to block Docker Hub typos. It was useful, but it was not provenance.

4.4 Signed by tag, verified by nobody. Cosign ran keyless on every build and signed tags. The documented verification command matched only the shared workflow’s identity at @refs/heads/main, which every caller on every branch produces. Nothing at admission ran it anyway. A separate “re-sign existing images when the build is skipped” path would have signed a forged image as soon as the next legitimate build of that SHA ran.

4.5 The scan gate that ran after the tags. The security doc said the image was scanned and blocked “before the convenience tags are pushed”. The workflow created the manifest tags first and scanned afterwards, so a failing scan left a fully tagged image in the registry. That is harmless only if nothing deploys by tag. Docs drift in exactly the claims auditors read. Check the order of steps, not the prose.

4.6 A lifecycle rule took production down 15 hours. A “keep the last 60 images” rule expired the tag production had been pinned to for weeks. At one repository’s push rate, 60 images was a 13-day window. Running pods were fine, because nodes had the image cached. The first pod scheduled onto a fresh node got ImagePullBackOff and wedged the rollout. A related variant broke a preview environment: a stable per-PR bootstrap tag fell past the cutoff because of churn on a different PR. The fix was age-based expiry with a 90-day floor, plus a scheduled pin-existence check for the mirror-image risk.

4.7 Partial multi-arch builds. About 6% of build groups had one architecture’s intermediate image and no manifest. That came from ordinary flakiness: OOM on one leg, a CodeBuild capacity hiccup. Making intermediates immutable would have converted every retry of those into a failure. It turned out to be unnecessary, because the manifest references children by digest.

4.8 A fork that rewrites its own release tag. A forked upstream project’s build re-created the <sha> manifest tag on every release push of a commit already built from main, and also maintained four channel tags. The repository couldn’t be made immutable without breaking releases, and the five-filter cap meant exclusions couldn’t express it either. The fix belongs in the build: don’t re-push a tag that exists. Until then, that repository is the softest target in the registry, so pin it by digest first.

4.9 Scanning the wrong architecture. trivy image repo:tag on an amd64 runner scanned only the amd64 child. A reviewer caught that arm64-only CVEs were invisible, and the fix was to scan each per-arch reference. Production on Graviton made this more than academic.

4.10 Twenty-one of twenty-two. The critical-CVE gate defaulted on and was turned off almost everywhere. A remediation sprint showed every critical finding across five repositories was base-image-level: OS patch lag, an unused bundled npm with a vulnerable tar, an esbuild Go binary. None was in application dependencies. The durable fixes were base-image hygiene and an expiring allowlist, not arguing about the boolean.

4.11 IAM size limits drive wildcard grants. Fifteen per-repo inline statements on one role exceeded 10,240 bytes. The quick fix, codebuild:StartBuild on project/*-amd64, let any trusted repository run any repository’s build with an overridden buildspec under that project’s service role. A reviewer narrowed it to the module’s naming convention, but the real fix is per-repo roles. When a quota forces a wildcard, treat that as a design signal.

4.12 OIDC subjects that changed shape. Newer repositories rendered the sub claim with numeric IDs, so the generated repo:org/name:* pattern never matched. Releases failed at “Assuming role”, and the tempting fix was a broader wildcard. The correct fix was a per-repo override, read from the GitHub API and recorded with a comment explaining how to read it.

4.13 Pull-through cache gotchas. Docker Hub 429s on shared NAT egress pushed one team to a pull-through cache. They then hit three surprises in one afternoon: the wrong secret key names, the CMK restriction, and the rule failing to create against an empty secret with an error that read like a wrong ARN.

4.14 Admission is pod-level, so GitOps looks green. After the Deny flip, the first blocked rollout looked like a stuck deployment. ArgoCD reported the sync succeeded, because the Deployment was admitted and only its pods were refused. A Terraform-managed Helm release timed out instead. Without the runbook entry, the on-call engineer would have debugged the wrong layer.

4.15 A Deny/Warn combination fails mid-apply. The API server rejects a binding with both Deny and Warn [8]. Put a variable validation in your module so it fails at plan time, not halfway through a production apply.

4.16 Credentials in the PR path. A PR-triggered check that verified production tags exist in ECR held id-token: write. That permission is granted per job, not per step, so PR-controlled code could have assumed a role that also read production secrets. The check was split into a credential-less PR job and a scheduled job with credentials. Supply-chain tooling is itself part of the supply chain.

5. Best-practice checklist

Day 1 (seed stage, roughly a week of work)

One ECR repository per deployable, IMMUTABLE_WITH_EXCLUSION, with exclusions limited to latest and caches. Nothing production uses may match an exclusion.
Remove ecr:BatchDeleteImage, DeleteRepository and PutImageTagMutability from every CI principal, in both identity and repository policies. Add an explicit Deny.
Use age-based lifecycle rules only, with a floor above the oldest production pin. Add a weekly pin-existence check.
Use GitHub OIDC with no static keys. Trust policies use StringEquals on exact sub values, and nothing trusted :* can push to a tier-0 repository.
Pin every third-party action and reusable workflow by full SHA, and enforce it with a linter.
Pass workflow inputs to shell through env:, never ${{ }} inside run:.
Run Trivy on each per-arch image in CI and block on critical fixable findings, with an expiring allowlist.
Turn on the ValidatingAdmissionPolicy registry allowlist in Audit/Warn mode on production.

First year (Series A)

Per-repo push roles trusted only on main or a protected environment, and a role_arn input in the shared workflow.
Per-repo CodeBuild service roles. StartBuild scoped to that repo’s projects.
Assemble multi-arch indexes from exported digests, then scan, sign and attest by digest.
Keyless cosign signing, with verification checking issuer, subject, source repository and ref.
SBOM (SPDX or CycloneDX) and SLSA L2 provenance attached as referrers.
Production values pin tag@sha256:digest. Update promotion automation and approval-exemption logic to match.
Admission allowlist names individual production repositories, and the VAP is flipped to Deny with a runbook entry and escape hatch.
Signature verification (Kyverno or policy-controller) on tier-0 namespaces in Audit, then Enforce.
Daily rescan of production-pinned digests, alerting for internet-facing services.
CODEOWNERS plus required code-owner review on the shared workflows repository and on .github/.
Debug and ops images mirrored into your own ECR and pinned in runbooks.

At scale (Series B and later)

Registry in a dedicated tooling account. Production accounts get pull-only grants, and an SCP denies history-rewrite actions outside break-glass.
Signature and attestation verification on every namespace, requiring a fresh vuln attestation for tier-0.
SLSA Build L3 for tier-0 via an isolated provenance generator. Hermetic, reproducible builds for signers.
Inspector continuous scanning, or equivalent, with findings mapped to running workloads and routed to Security Hub.
SBOM inventory you can query during incidents (“which running digests contain package X version Y?”).
Verify upstream signatures for base images (Chainguard, Docker Official Images, distroless) before mirroring.

6. Maturity roadmap

Stage	What exists	Write set for prod `signer-svc`	Next step
0: Ad hoc	Static keys, mutable `latest`, deploy by tag	Anyone with the key, plus anyone who can push	OIDC, immutability, delete removed
1: Registry hygiene	OIDC (shared role), immutable tags, age-based lifecycle, Trivy in CI	Every branch and PR of every trusted repo	Per-repo `main`-only roles
2: Scoped identity	Per-repo roles, per-repo CodeBuild roles, SHA-pinned workflows, VAP allowlist in Deny	Merged code in `signer-svc` plus the shared workflow’s maintainers	Digest everywhere, keyless signing, claims-aware verification
3: Verified provenance	Sign and attest by digest, prod pinned by digest, signature enforcement on tier-0	Merged code in `signer-svc`, provably	Separate registry account, SLSA L3, fleet-wide enforcement
4: Hardened factory	Isolated provenance, hermetic builds, multi-account registry, continuous scanning with SBOM inventory	Merged code, built by an isolated builder	Reproducible builds, independent rebuild verification for signers

Most startups can reach Stage 2 in a quarter with one engineer, and Stage 3 for tier-0 services in the following quarter. Order the work by the write set: remove delete, scope the roles and narrow the allowlist before investing in signatures. A signature that verifies an identity anyone can assume is a more elaborate version of the same hole.

References

AWS. Preventing image tags from being overwritten in Amazon ECR. https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html
AWS. Automate the cleanup of images by using lifecycle policies in Amazon ECR. https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html
OpenSSF. SLSA v1.0: Security levels. https://slsa.dev/spec/v1.0/levels
GitHub. About security hardening with OpenID Connect. https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect
AWS. Create a role for OpenID Connect federation: Configuring a role for GitHub OIDC identity provider. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html
GitHub. Security hardening for GitHub Actions (script injection, pinning, reusable workflows). https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions
Sigstore. Fulcio certificate extension OIDs. https://github.com/sigstore/fulcio/blob/main/docs/oid-info.md
Kubernetes. Validating Admission Policy. https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
Sigstore. Policy Controller overview. https://docs.sigstore.dev/policy-controller/overview/
Kyverno. Verify Images: overview (mutateDigest, verifyDigest, required). https://kyverno.io/docs/policy-types/cluster-policy/verify-images/overview/
GitHub. Using artifact attestations to establish provenance for builds. https://docs.github.com/en/actions/security-for-github-actions/using-artifact-attestations/using-artifact-attestations-to-establish-provenance-for-builds
Docker. Multi-platform builds. https://docs.docker.com/build/building/multi-platform/
AWS. Sync an upstream registry with an Amazon ECR private registry (pull through cache). https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html
AWS. Scanning Amazon ECR container images with Amazon Inspector. https://docs.aws.amazon.com/inspector/latest/user/scanning-ecr.html
Sigstore. Signing containers with cosign. https://docs.sigstore.dev/cosign/signing/signing_with_containers/
CISA. Supply Chain Compromise of Third-Party tj-actions/changed-files (CVE-2025-30066) and reviewdog/action-setup (CVE-2025-30154). https://www.cisa.gov/news-events/alerts/2025/03/18/supply-chain-compromise-third-party-tj-actionschanged-files-cve-2025-30066-and-reviewdogaction
Elastic Security Labs. Bit ByBit: emulation of the DPRK’s largest cryptocurrency heist. https://www.elastic.co/security-labs/bit-bybit
TechCrunch. Supply chain attack targeting Ledger crypto wallet leaves users hacked (Dec 2023). https://techcrunch.com/2023/12/14/supply-chain-attack-targeting-ledger-crypto-wallet-leaves-users-hacked/
AWS Well-Architected Framework, Security Pillar. SEC11-BP06 Deploy software programmatically. https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/sec_appsec_deploy_software_programmatically.html
NIST. SP 800-218: Secure Software Development Framework (SSDF) Version 1.1 (Feb 2022). https://csrc.nist.gov/pubs/sp/800/218/final
CNCF TAG Security. Software Supply Chain Best Practices. https://tag-security.cncf.io/community/working-groups/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf
NIST. SP 800-190: Application Container Security Guide (Sep 2017). https://csrc.nist.gov/pubs/sp/800/190/final