Building on AWS · guide 04 of 13

GitOps and Progressive Delivery on EKS with Argo CD and Argo Rollouts

6853 words · about 35 minutes · all guides

Abstract. This guide is for senior platform, DevOps and SRE engineers at startups that run Kubernetes on Amazon EKS and are adopting, or have already adopted, Argo CD and Argo Rollouts. It is written with crypto-infrastructure teams in mind: relayers, signers, indexers and RPC proxies, where a bad deploy can mean stuck user funds, missed settlement windows or a signing service that silently stops signing. It is not a tutorial. It assumes you know what an Application and a Rollout are. It covers the decisions that decide whether your GitOps setup is a safety system or just a slower kubectl apply:

the hub-cluster topology;
how Applications get generated: Terraform, ApplicationSets or app-of-apps;
AppProjects, and why leaving everything in default is a latent privilege escalation;
sync policy semantics, including the one rule that leaves apps stuck forever;
sync waves, and a first-install deadlock between the HPA and the Deployment;
multi-source $values and the “two merges” class of outage;
promotion through bot PRs and prod approval gates;
blue/green vs canary, and analysis that actually gates;
automatic git-revert rollback;
coexisting with Terraform-managed helm_releases;
alerting on the failure mode that nothing else catches: an Application that is OutOfSync but reads Healthy.

The lessons come from operating a mid-sized platform: roughly 30 services, around 90 Argo CD Applications across dev and prod, and a team of a few people. Each failure described here happened at least once.

1. The problem space

1.1 Forces specific to startups

A startup platform team is usually two to five people supporting fifteen to fifty services. The forces are predictable:

Deploy frequency is high and uneven. In one platform we studied, image promotions made up about three quarters of all pushes to the infra repo’s main. Every friction point in the deploy path is paid many times a day, and a human gate that adds ten minutes adds hours of engineer time per week.
The team can’t watch dashboards during every rollout. If the system does not decide by itself whether a release is safe, nobody does.
External assurance pressure arrives early. Exchange partners, institutional customers and SOC 2 auditors will ask about change management: who approved this prod change, and how do you roll back? AWS Well-Architected frames this as REL08-BP05, “Deploy changes with automation” [19]. GitOps gives you a strong answer for free, if the git history really is the record of what ran.
Cost pushes toward fewer clusters. A separate Argo CD per environment costs nodes, upgrades and attention. A single hub is cheaper, and it concentrates risk.

1.2 Crypto-specific stakes

Web3 infrastructure turns ordinary deploy risk into different shapes of risk:

Signers and key-holding services. A misconfigured deploy that changes a signer’s KMS key reference, IAM role or allowed-origin list can be worse than an outage. An outage is visible. A signer that signs for the wrong policy is not.
Relayers and settlement paths. A relayer that is “Healthy” (its readiness probe passes) but cannot reach its RPC provider, or has a nonce manager stuck on a bad sequence, fails functionally while every Kubernetes signal is green. Probe-based promotion is not enough.
Indexers with long warm-up. Blue/green looks attractive until the new color needs 40 minutes to backfill before it can serve. Canary-by-replica-count may be the only realistic progressive strategy.
Hot-path latency. Quoting and routing services have tight latency SLOs. A canary that passes on error rate but adds 300 ms of p99 has still broken the product.

All of this argues for functional analysis (end-to-end tests that move real, if tiny, value on a testnet or a mainnet canary account) over purely infrastructural health signals.

1.3 What GitOps promises, and what it does not

The OpenGitOps principles are: declarative, versioned and immutable, pulled automatically, continuously reconciled [1]. A naive reading is that git is the cluster. In practice there are at least four gaps, and each is a section of this guide:

Git can describe a state the controller will never reach. A failed sync for a given commit is not retried by default [2].
Two systems can each own half of an Application’s desired state. For example, Terraform owns the list of values files and git owns their contents, and the cluster passes through a state that neither system describes.
Rendered manifests and plans describe end states, not transitions. A clean diff is not evidence that the change is safe to roll out.
“Progressive delivery” that is configured is not the same as progressive delivery that gates. An empty canary is a rolling update with a nicer name [12].

2. Reference design

2.1 Topology

From a service repository's image build to an environment's running version: the CD workflow, the bot pull request, and the promotion path. full size ↗

The design choices:

One Argo CD on a dedicated hub (“tools”) cluster manages dev and prod remotely. Dev and prod run only the Argo Rollouts controller and the workloads. The hub also hosts observability, so the system that deploys and the system that watches deploys share one failure domain, and neither shares it with production.
Hub-to-spoke API access over private networking. If your VPC CIDRs overlap (they often do when every environment was stamped from one module default), use an interface VPC endpoint (PrivateLink) to the spoke’s EKS API, plus a private hosted zone that overrides the API hostname, so TLS verifies without a --server override. Don’t open the spoke API to the internet for the hub’s benefit; the EKS Best Practices Guide covers cluster-endpoint access in depth [22].
The EKS managed Argo CD capability is now an option [21]. The controllers run in the AWS control plane and integrate with IAM Identity Center. Evaluate it before you self-host. You give up some control over controller flags and plugin sidecars, and you stop owning Argo CD upgrades.

2.2 Repo layout and the four-layer values stack

helm-charts/
  common/                    # library chart: Deployment|Rollout, HPA, PDB, NetPol, AnalysisTemplate
  payments-api/              # one-line wrapper: dependency on common + chart defaults
argocd/apps/
  values-dev.yaml            # environment-wide (arch, OTel endpoint, node selectors)
  values-prod.yaml
  payments-api/
    values-common.yaml       # this app, every environment (opt-in)
    values-dev.yaml          # this app, dev   <- bot edits image.tag here
    values-prod.yaml         # this app, prod  <- bot edits image.tag here

Precedence, from least to most specific: chart defaults, then environment-wide values, then the app’s values-common.yaml, then the app’s per-environment file. This stack is worth having. Without the values-common layer, everything true for a service in both environments gets written twice, and an accidental dev/prod difference becomes indistinguishable from a deliberate one. In one audit, 43% of the leaf keys present in both dev and prod files were byte-identical duplicates.

Make the common layer opt-in per app rather than relying on ignoreMissingValueFiles. That flag would also silence a typo’d or renamed values-prod.yaml, and the app would quietly sync on chart defaults instead of failing.

2.3 The Application, generated by Terraform

resource "kubernetes_manifest" "prod" {
  count           = var.create_prod ? 1 : 0
  computed_fields = ["metadata.annotations", "metadata.labels", "operation"]
  field_manager { force_conflicts = true }

  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name       = "${var.service_name}-prod"
      namespace  = "argocd"
      finalizers = ["resources-finalizer.argocd.argoproj.io"]
      annotations = {
        "notifications.argoproj.io/subscribe.on-sync-failed.slack" = "deploys"
      }
    }
    spec = {
      project = var.project          # NOT "default" -- see section 3.2
      sources = [
        { repoURL = var.repo_url, targetRevision = "HEAD", ref = "values" },
        {
          repoURL        = var.repo_url
          targetRevision = "HEAD"
          path           = var.chart_path
          helm = {
            releaseName = var.service_name
            valueFiles = compact([
              "values.yaml",
              "$values/argocd/apps/values-prod.yaml",
              var.common_values ? "$values/argocd/apps/${var.service_name}/values-common.yaml" : "",
              "$values/argocd/apps/${var.service_name}/values-prod.yaml",
            ])
          }
        },
      ]
      destination = { name = "prod", namespace = var.namespace }
      syncPolicy = {
        automated   = { prune = true, selfHeal = true }
        syncOptions = ["CreateNamespace=false"]
        retry = {
          limit   = 5
          backoff = { duration = "10s", factor = 2, maxDuration = "3m" }
        }
      }
    }
  }
}

Three details in this block come straight from incidents.

computed_fields must list operation, and field_manager.force_conflicts must be true. The Terraform kubernetes_manifest resource resubmits the whole field set it last read [28]. Argo CD’s own controllers write .operation.* and notification annotations onto the same object. Without these two settings you get “Provider produced inconsistent result after apply” on fields you never declared, or a write the API server rejects outright. Setting computed_fields replaces the provider default (annotations and labels) rather than extending it, so restate those defaults too. One more trap: once force_conflicts has been applied to objects with a live conflict, removing it again is a write the conflict itself blocks. Terraform then re-plans that impossible removal on every run.
The resources finalizer must be present before you delete. Without it, deleting an Application leaves the workload orphaned and running. The finalizer has to be on the live object for a Terraform destroy to cascade. Applications created before you added it need a one-time argocd app delete --cascade.
Don’t declare selfHeal = false. The CRD field is omitempty. The API server drops an explicit false, so every plan shows a permanent null -> false diff. That kept one team’s Argo CD Terraform root “dirty” on every run, and two concurrent applies then raced for its state lock. Omit the key when you want self-heal off.

2.4 The library chart: one switch, safe ordering

# values.yaml (per app)
rollout:
  enabled: true
  strategy:
    canary:
      canaryService: payments-api-canary
      stableService: payments-api
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: payments-api-sli }] }
        - setWeight: 50
        - pause: { duration: 10m }
  analysis:
    sliMetrics:
      enabled: true
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 12

The library renders a Rollout when rollout.enabled is set and a Deployment otherwise, and the HPA’s scaleTargetRef follows the switch. It also encodes the sync order in annotations:

Wave	Resources
`-1`	Secret material (external-secrets / 1Password / Secrets Manager CRs)
`0`	ServiceAccount, Service, ConfigMap, NetworkPolicy, migration Job
`1`	Deployment / Rollout
`2`	HPA

Section 5.2 explains why the HPA must be in a later wave than the workload.

3. Decisions and tradeoffs

3.1 How Applications are created

Option	Upsides	Downsides	Good for
App-of-apps (a root Application pointing at a directory of Application YAML) [6]	Pure GitOps; bootstraps a cluster from one object; reviewable YAML	Boilerplate per app and environment; drift between hand-written Applications; the root app becomes a high-blast-radius object	Bootstrapping platform add-ons; teams with no Terraform
ApplicationSet (generators: list, cluster, git directory, pull request, matrix) [7]	One template, many apps; PR-preview environments for free; adding a directory adds an app	Template logic is a new language (`goTemplate`, `missingkey=error`); a bad template edit hits every generated app at once; ownership is subtle when something else edits generated apps	Uniform fleets; per-PR previews; many clusters
Terraform-generated (`kubernetes_manifest` per app, from a module)	Same review and plan workflow as the rest of infra; Application wiring can reference Terraform outputs (cluster endpoints, IRSA role ARNs, secret names); one module encodes policy (sync policy, finalizers, ignore rules)	Two systems own one object (see 2.3); the Application spec changes only on a Terraform apply, which creates the two-merge problem (5.4); slow applies once you pass about 100 Applications	Teams already running everything through Terraform, with fewer than about 150 Applications

Recommendation by stage:

Seed: app-of-apps, or a handful of hand-written Applications (see also the Argo CD best-practices page on separating config from source repos [11]). Don’t build a module for five services.
Series A: Terraform-generated, if Terraform already owns the clusters and IAM. Wiring Applications to IRSA roles and cluster outputs in the same plan is a real advantage. Use an ApplicationSet with a pull-request generator for preview environments.
Scale (more than ~150 Applications, multiple clusters per environment, a dedicated platform team): move the bulk to ApplicationSets with cluster and git generators, and keep Terraform for the hub itself, AppProjects and cluster registration.

Signal to revisit: Argo CD Terraform applies take more than five minutes; you hit the kubernetes_manifest conflicts above more than once a quarter; or app teams wait on platform reviews just to add an app.

ApplicationSet ownership gotcha. When something other than the ApplicationSet controller writes to generated Applications, applicationsSync: create-only is not enough. A trusted preview deployer that pins an image digest is one example. The controller will still revert the fields it considers its own. You need ignoreApplicationDifferences and preservedFields for the paths the other writer owns [7].

3.2 AppProjects: the `default` project is a privilege boundary you never drew

The default AppProject permits any source repo, any destination cluster and namespace, and every resource kind, including cluster-scoped ones [5]. If every Application lives in default, then anyone who can merge a values change can render a ClusterRoleBinding into prod. An independent review of the platform described here called this out explicitly. It is the most common Argo CD finding we see, and it matters more for crypto teams because the same cluster runs the signers.

A minimal production project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod-workloads
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example-org/infra.git
  destinations:
    - name: prod
      namespace: workloads
  clusterResourceWhitelist: []          # no cluster-scoped kinds at all
  namespaceResourceBlacklist:
    - group: rbac.authorization.k8s.io
      kind: RoleBinding                 # bindings come from the platform project only
    - group: ""
      kind: ResourceQuota
  roles:
    - name: deployer
      policies:
        - p, proj:prod-workloads:deployer, applications, sync, prod-workloads/*, allow
      groups: [platform-oncall]
  syncWindows:
    - kind: deny
      schedule: "0 22 * * 5"            # optional: no automated syncs Fri 22:00 for 60h
      duration: 60h
      applications: ["*"]
      manualSync: true

Split projects at least into platform (CRDs, controllers, cluster RBAC; few reviewers) and <env>-workloads (namespaced kinds only). Put key-holding services in their own project with a tighter destination namespace, so a values PR for the indexer cannot render into the signer’s namespace.

Cost: an hour of work, plus the occasional “why can’t my chart create a ClusterRole” ticket. That friction is the point.

3.3 Sync policy

The semantics people get wrong are documented plainly [2]:

Automatic sync will not reattempt a sync if the previous sync attempt against the same commit-SHA and parameters had failed.

selfHeal doesn’t help here, because it reconciles drift after a successful sync. So a transient failure leaves the Application OutOfSync on the old spec with health still Healthy, until a human syncs or an unrelated commit changes the SHA. Examples of transient failures: an API server hiccup, an admission webhook timeout, or a multi-source ComparisonError because main moved mid-apply. Nothing about the Application looks broken, which is why this failure is so dangerous.

The knobs:

Setting	Recommendation	Why
`automated.prune`	`true`	Without it, deleting a resource from git leaves it running indefinitely. With it, an accidental deletion in git deletes in prod. Pair it with review and `Prune=false` annotations on PVCs and stateful objects [9].
`automated.selfHeal`	`true` in prod; your choice in dev	Dev and preview apps often sync from unreviewed branches and get hand-edited during debugging. Self-heal there fights engineers. In prod, live drift is an incident.
`retry`	`limit: 5`, `10s × 2` up to `3m`	Bounded retries cover transient failures without hammering a broken webhook. The total window is about 8–10 minutes.
`retry.refresh`	`true` on recent Argo CD versions	A retrying sync picks up a newer revision instead of retrying a stale one [2].
`RespectIgnoreDifferences`	Only where needed	See 5.6. Turning it on globally changed client-side apply behavior and stopped one team’s Rollouts from advancing their image.

Signal to revisit: you find Applications sitting OutOfSync for hours after an infra merge, or deploy tooling that waits on Synced times out on deploys that actually succeeded.

3.4 Promotion: bot PRs editing image tags

Options:

CI pushes directly to main. Fast, but no review artifact and no clean revert unit.
Argo CD Image Updater, or Flux image automation. Little glue code, but the “what’s in prod” decision moves into a controller’s tag-selection rules, and gating on dev health or human approval gets awkward.
A CD workflow opens a bot PR per environment that changes only image.tag, then merges it. It is explicit and revertable, and every promotion is a commit you can git log.

Option 3 is the right default for a small team. The pattern:

# Sketch of a CD workflow (GitHub Actions)
concurrency:
  group: promote-${{ inputs.env }}-${{ inputs.app }}
  cancel-in-progress: false               # never cancel a deploy halfway
jobs:
  deploy-dev:
    uses: ./.github/workflows/promote.yaml
    with: { env: dev, app: ${{ inputs.app }}, sha: ${{ inputs.sha }} }
  await-prod-approval:
    needs: deploy-dev
    steps:
      - run: ./scripts/open-approval-issue.sh   # issue + chat ping
      - run: ./scripts/wait-approval.sh         # polls for an exact "approved"/"denied" comment
        timeout-minutes: 60                     # no decision = no deploy (fail closed)
  deploy-prod:
    needs: await-prod-approval
    uses: ./.github/workflows/promote.yaml
    with: { env: prod, app: ${{ inputs.app }}, sha: ${{ inputs.sha }} }

Make these properties non-negotiable:

Tags are immutable and unique. Use the git SHA, and enable ECR tag immutability on every repository, including the per-architecture intermediates of multi-arch builds. Pinning a digest is stronger. If you pin tags, the ECR immutability setting is the whole integrity story.
Classify promotions by diff, not by path. If promotions skip an approval gate, decide that from the diff: every changed line must be one of the keys the automation writes (image.tag, a deploy annotation, a nonce). The same values file also holds env vars, secret references, NetworkPolicy and resource limits, and Argo CD applies a config-only edit there straight from main. A path-based exemption is a bypass. Removing an analysis-skip flag can be exempt; adding one never should be.
Newer runs supersede older ones. If two promotions for the same app are in flight, the older one’s approval should be withdrawn, and its wait loop should treat “main now pins a different tag” as a no-op success rather than a ten-minute timeout.
Success means “target image live and Healthy”, not “Synced”. In a busy monorepo, HEAD moves past your commit constantly, and the app reads OutOfSync while running exactly the right image. Match the tag against .status.summary.images. For changes that don’t move the tag (config-only hotfixes, toggling an analysis flag), wait for the last synced revision to reach your commit instead.
Dispatch inputs go through env:, never ${{ inputs.x }} inline in run:. This is GitHub’s own hardening guidance [27]. CD workflows run with deploy credentials, and inline interpolation is a script-injection sink for anyone with write access to a service repo that can dispatch them.

3.5 Approval gates for prod

GitHub Environment required reviewers are the obvious control, but on private repos they need a higher-tier plan than many startups pay for. The setup call fails, and some teams have closed an “approval gate” ticket without noticing it was never configured. A workable alternative is an issue-comment gate:

The workflow opens an issue and pings chat.
It polls for a comment that is exactly approved or denied from an allow-list of approvers.
It parses comments as structured objects (author, body, timestamp) with a tested jq filter.
It fails closed after an hour.

Be honest about what this buys. It is workflow code, not a platform control: anyone who can edit the workflow can remove it, and one person can often approve their own change. It gives you an audit trail and a deliberate act, not separation of duties. To get separation of duties, put the workflow directory and the gate scripts under required code-owner review in a branch ruleset.

Two more points:

Count every path to prod. A workflow_dispatch of the per-environment promote workflow often bypasses the orchestrating pipeline’s gate. Treat it as break-glass and say so in the runbook.
Use an allow-list for exemptions. Some services (low-risk frontends, preview tooling) may legitimately skip prod approval. Keep that list explicit in code, not implicit in naming.

3.6 Blue/green vs canary

	Blue/green [13]	Canary [12]
Mechanism	Full new ReplicaSet behind a preview Service; switch the active selector; keep old for `scaleDownDelaySeconds`	Shift traffic or replicas in steps
Cost	2× pods during the rollout	+1 pod (replica-based) or a small slice (traffic-routed)
Best gate	`prePromotionAnalysis` (test the preview Service) or `postPromotionAnalysis` (test live, abort switches back)	Inline `analysis` steps between weights, plus background analysis
Rollback speed	Seconds (selector flip), while the old RS is still up	Seconds to minutes
Fits	Stateless APIs where an E2E suite can drive the preview endpoint; services where mixed versions are dangerous (protocol or nonce changes)	High-traffic services where SLI deltas are statistically meaningful; services where 2× capacity is expensive (indexers, GPU)
Poor fit	Workers consuming from queues (both colors consume); long warm-up	Low-traffic services, where 10% of almost nothing gives no signal

Notes that matter for crypto services:

Relayers and nonce managers. Two versions submitting transactions from the same EOA at once is a correctness problem, not just a compatibility one. Blue/green with a short scaleDownDelaySeconds, or a single-writer lease, beats a canary.
Queue consumers. Both a canary and a blue/green preview will consume from RabbitMQ or SQS. Without traffic routing at the broker, “10% of traffic” is meaningless. Gate on functional E2E instead.
Without a traffic router, a canary’s weight is approximated by replica count [12]. With 3 replicas, setWeight: 10 really means one pod in four. Use a traffic router (ALB, Traefik, a mesh) or accept the coarseness.

Martin Fowler’s original descriptions [23] still frame the tradeoff well. The Google SRE Workbook chapter on canarying [17] gives the statistical argument for canary population size and duration.

3.7 Analysis: E2E jobs vs SLI queries

Argo Rollouts supports several analysis providers, including Prometheus-compatible queries and Kubernetes Jobs [14][15][16]. Use both, for different questions.

E2E Job analysis answers “does the new version work?”:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: payments-api-e2e }
spec:
  args: [{ name: deploy-image }]
  metrics:
    - name: e2e
      provider:
        job:
          spec:
            backoffLimit: 0
            activeDeadlineSeconds: 900
            template:
              spec:
                restartPolicy: Never
                securityContext: { runAsNonRoot: true, seccompProfile: { type: RuntimeDefault } }
                containers:
                  - name: e2e
                    image: registry.example/payments-api-e2e:{{args.deploy-image}}
                    env:
                      - { name: TARGET_URL, value: http://payments-api-preview }

SLI analysis answers “is the new version worse?”:

  metrics:
    - name: error-ratio
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://vmselect.observability:8481/select/0/prometheus
          query: |
            sum(rate(http_server_requests_total{service="payments-api",
                     rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[2m]))
            /
            sum(rate(http_server_requests_total{service="payments-api",
                     rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))

Hard-won rules for SLI analysis:

Scope the query to the canary’s pod-template hash, or compare canary against stable. A query over the whole service dilutes a 100% failure on one canary pod into a 25% blip.
Decide what an empty result means. A query that returns no series, because the label was renamed or the scrape job is missing, must not count as success. Write successCondition so that an empty result fails, or the analysis turns into a rubber stamp the first time someone renames a metric.
Know your environment-label conventions. Many stacks end up with several incompatible ways of tagging env. A templated placeholder substituted per environment is safer than a hard-coded label value.
The hotfix escape hatch must be explicit and self-clearing. A skipAnalysis value that strips the analysis blocks, set together with a nonce annotation that forces a new pod-template hash, gives you an emergency path. The next normal deploy clears both. An aborted rollout sticks to its revision, and an unchanged template starts no new one, so a “retry with the same SHA” also needs the nonce, this time without skipAnalysis, so the gate runs again. CI usually can’t reach the workload cluster to run kubectl argo rollouts retry.

3.8 Terraform `helm_release` alongside Argo CD

Some releases legitimately stay in Terraform:

ingress controllers whose Service creates an NLB that Terraform also references;
OTel collectors wired to Terraform-created IAM roles;
anything whose values are mostly Terraform outputs.

That is a valid choice, but name the costs:

“Merged” means different things for different releases. For a helm_release, the deploy is the Terraform apply run, so verify with helm_release.<name>: Modifications complete in the apply log. kubectl get applications will never list it.
Namespace tells you nothing about ownership. A dozen Terraform-managed releases can sit in the same namespace as Argo CD apps. Keep a derivable inventory (grep -l 'namespace *= *"workloads"' envs/prod-apps/*.tf) and document it.
Chart edits must trigger the apply. Put helm-charts/** in the apply workflow’s path filter, and feed a chart checksum into the release so a template change forces a diff.
The promotion path forks. Your CD workflow needs a second branch for these releases: a targeted Terraform apply that includes a rollout watch, with the PR merged only after the apply succeeds. A targeted apply that runs outside the main apply’s concurrency group can collide on the state lock. -refresh=false speeds it up but hides drift.

Signal to revisit: more than about 15% of your workloads are helm_releases, or two people have independently asked “why didn’t my chart change deploy?”

4. Rollback

4.1 Two layers that must agree

A failed rollout has two separate states to repair:

Live pods. Argo Rollouts aborts to the stable ReplicaSet within seconds when analysis fails.
Git desired state. main still pins the bad SHA, because the merge is what triggered the sync. Argo CD’s desired state is still bad, and the next sync, a self-heal or a manual “sync” by a confused on-call engineer redeploys it.

So the CD workflow must re-commit the previous tag automatically when the rollout fails. The promote job knows the old tag from its own diff. If the wait step fails and the tag actually changed, it opens and merges a revert PR so that git matches the stable ReplicaSet again:

- name: Revert image tag on deploy failure
  if: >-
    always() && steps.wait.outcome == 'failure'
    && steps.bump.outputs.old_tag != ''
    && steps.bump.outputs.old_tag != inputs.sha
  env:
    APP: ${{ inputs.app }}
    ENV: ${{ inputs.env }}
    TAG: ${{ steps.bump.outputs.old_tag }}
  run: ./scripts/create-and-merge-tag-pr.sh

The Terraform path needs no equivalent, because its merge is already gated on the apply, including the rollout watch, succeeding.

4.2 Rollback matrix

Scenario	Mechanism	Typical time
Analysis fails during rollout	Rollouts aborts to stable; CD re-commits the old tag	1–3 min
Rollout never becomes Healthy (no analysis)	Wait step times out; CD re-commits the old tag	Wait timeout + one sync (often 10+ min)
Regression found after promotion	Dispatch promote with the previous SHA, flagged as rollback	1–2 min plus approval
Emergency fix	Hotfix dispatch: new SHA plus `skipAnalysis` plus nonce	~2 min

The second row is why empty canaries are expensive. Without analysis, your rollback trigger is a timeout, and your detection is “pods eventually didn’t go Ready”, which a functionally broken relayer never trips.

5. Failure modes and field lessons

5.1 The stuck-but-Healthy Application

What happened. An infra PR merged. During the apply, main moved, and the multi-source Application hit a ComparisonError between a pinned revision and HEAD. The sync failed once. Because Argo CD doesn’t retry a failed sync for the same SHA [2], and self-heal was off, the app sat OutOfSync on the old spec for over 30 minutes. Health read Healthy the whole time. Nobody noticed until a downstream symptom paged.

Fixes, in layers:

retry with bounded backoff, so transient failures self-resolve.
selfHeal: true in prod.
An alert on the residual (5.8).
Runbook knowledge: a hard refresh (argocd.argoproj.io/refresh=hard) clears a stale ComparisonError but does not make the app converge. You still need to sync manually or land a new commit.

5.2 The HPA-before-Deployment first-install deadlock

Argo CD applies one wave at a time and holds the next wave until every resource in the current one is healthy [3]. Its docs warn that an app with unhealthy resources in an early wave “can never get to healthy.” An HPA whose scaleTargetRef doesn’t exist yet reports Degraded (FailedGetScale) [24]. So if the HPA shares wave 0 with the Service and ServiceAccount, and the Deployment sits in wave 1, a first install deadlocks: everything else lands, the HPA waits for a Deployment the sync will never apply, and no number of new revisions clears it.

The nasty part is that only fresh installs are exposed. Apps whose Deployment predated their HPA never hit it, so the bug survived across seven or so autoscaled services before a new service’s first install exposed it. The fix is one annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"   # strictly after the workload (wave 1)

Leave a comment in the template explaining why, because “tidy up the wave numbers” is a tempting refactor. The general lesson: any resource whose health depends on another resource must be in a strictly later wave. The same applies to PDBs with selectors, ServiceMonitors and KEDA ScaledObjects.

5.3 Multi-source `$values` pitfalls

Multi-source Applications let a chart in one place read values from anywhere in the repo through ref: values [4]. Gotchas:

$values must be the first path segment, and it resolves relative to the root of the referenced source [4].
If the ref source also has a path, Argo CD tries to render manifests from it as well. Leave path off the values source.
Both sources tracking HEAD doesn’t make them atomic. They are resolved separately. A pinned-SHA vs HEAD mismatch during an apply produces a ComparisonError (see 5.1).
A missing value file fails the sync, which is what you want (2.2). It also means every valueFiles change and every file move is an ordering problem (5.4).

5.4 The “two merges” problem

When Terraform owns the Application’s valueFiles list and git owns the values contents, a single PR that changes both creates a window where the cluster renders a state neither side describes.

What happened. A PR moved shared keys from values-prod.yaml into a new values-common.yaml and set common_values = true in the Terraform module, all in one merge. Argo CD reads values files from HEAD the moment the PR merges, so the keys disappeared from values-prod.yaml immediately. The Application’s valueFiles list didn’t include values-common.yaml until the Argo CD Terraform apply ran, minutes later and queued behind other environments in a max-parallel: 1 matrix. For about six minutes, a required env var belonged to no file the Application was reading. The service crash-looped in prod.

Neither the rendered-manifest diff nor the Terraform plan shows this. Both describe the end state. A clean plan and an empty Helm diff are not evidence of safety for this class of change.

The fix is procedural, with CI support:

Merge 1: add values-common.yaml and set the flag, leaving values-prod.yaml untouched. The duplicate keys make this a no-op render, safe on its own.
Wait for the Terraform apply.
Merge 2: delete the duplicated keys from the per-environment file.

Then teach your Helm-diff CI to render the transition state explicitly: the new values contents with the old valueFiles list. Fail the PR if that render differs from the end state in a way that drops keys. Few platforms do this, and it pays for itself the first time it fires.

Generalize it. Any time two systems with different apply latencies each own part of one object, the merge order matters and the diff tools are blind to it. The same shape shows up in other places:

an alert rule merged alongside the scrape job it depends on (the rule pages on NoData before the metric exists);
a config file and the env var it expands, applied by different pipelines;
an ingress port move, where the target group re-registers onto pods that aren’t listening yet.

Write each of these down as an explicit multi-merge procedure.

5.5 Empty canary steps as theater

One audit found 21 charts with Rollouts enabled. Every canary had steps: [], and SLI analysis was enabled nowhere. Per the Argo Rollouts docs, a canary with no steps “will mimic the rolling update behavior” [12]. The team was paying the operational cost of Rollouts (a second controller, ignoreDifferences for replicas, kubectl argo rollouts muscle memory, Rollout-aware HPA wiring) and getting a Deployment.

A further trap: an explicitly empty steps: [] gets dropped by the API server on write, so git and live never match, and the Application sits permanently OutOfSync unless you add an ignoreDifferences entry for /spec/strategy/canary/steps. That ignore rule then hides real drift for any app that later adds real steps, so scope it per app.

Be honest in your architecture doc. Write “only service X has a real gate; every other Rollout is an ungated rolling update” rather than “we use canary deployments.” Then pick the two or three services where a bad deploy costs money (signers, relayers, quoting) and give them real analysis before you touch the rest. If a service will never get analysis, render it as a plain Deployment and remove a moving part.

5.6 `ignoreDifferences` is display-only unless you opt in

ignoreDifferences suppresses the OutOfSync status, but a sync still applies the full desired manifest, ignored fields included, unless RespectIgnoreDifferences=true is set [8][9]. This bites on immutable fields: a bound PVC’s spec.volumeName is written by the provisioner, the chart never declares it, and every sync tries to patch it back to empty and fails. You need both the ignore rule and the sync option, scoped to the apps that need them. One team that enabled the option globally found that client-side apply started reporting their Rollouts as “unchanged” and stopped advancing the image on every later deploy.

Also: an ignoreDifferences entry with no group matches only the core (empty) API group, so kind: Deployment without group: apps silently matches nothing.

5.7 Moving content between values files, and other “end-state diffs lie” cases

The two-merge rule (5.4) has relatives:

Library-chart edits. If your common library is referenced by file:// and every Application tracks HEAD, a library edit reaches dev and prod for every app in one merge. There is no library-level staging. Either version the library and bump it per environment, or make Helm-diff CI render every dependent app on any library change (cheaper, and sufficient at small scale).
HPA min/max changes during a rollout. The Rollouts controller owns replicas mid-step. Without an ignoreDifferences on /spec/replicas, Argo CD resets the count on every sync.
Migration Jobs that carry the Service selector labels. While the Job runs, its pod is a Ready endpoint of the app’s Service and receives traffic on a port nothing listens on. Give hook Jobs their own label set.

5.8 Alerting on OutOfSync

Argo CD exports argocd_app_info with sync_status and health_status labels [10]. The alert that catches 5.1:

- alert: ArgoAppStuckOutOfSync
  expr: argocd_app_info{sync_status="OutOfSync", dest_server=~".*prod.*"} == 1
  for: 30m
  labels: { severity: sev2 }
  annotations:
    summary: "{{ $labels.name }} OutOfSync for 30m+"
    runbook_url: https://runbooks.example/argocd#stuck-outofsync

Field notes:

Scope it to prod. Dev and preview apps sync from unreviewed branches all day. A team that alerted fleet-wide got paged sev2 for a dev test app within days and scoped it back down.
Absent means “nothing out of sync”. The metric only emits a series for apps in that state, so treat no data as OK.
30 minutes is roughly right. It is longer than the retry budget (about 10 minutes) plus a slow Rollout, and shorter than “someone notices on their own.”
Alert on sync failures sooner, through notifications. Subscribe every Application to on-sync-failed. Be wary of subscribing everything to on-deployed: with self-heal and a busy repo it becomes noise people mute.
Pair it with a Rollout-level signal. An Application can be Synced while its Rollout is Degraded or paused on a failed analysis. Alert on Rollout phase too.

5.9 Retry budget expiry after a half-landed infra merge

Even with retries and self-heal, an infra merge that half-lands can outlast five attempts. One example is a CRD change applied by Terraform after the Argo CD sync that needs it. Make “check kubectl get applications -n argocd for OutOfSync” part of the post-merge checklist for any infra change, not just app deploys. Add a scripted “verify merge” step that confirms the change at the API level: the rule exists, the image is live, the Application synced the expected revision.

5.10 Deploy tooling and observability failing together

If the hub cluster that runs Argo CD also runs your metrics stack, a hub incident takes out both deploys and the analysis queries those deploys depend on. Analysis then fails, or worse, succeeds on empty data (3.7). Decide explicitly what happens when the metrics backend is unreachable: inconclusive should pause the rollout, never promote it. Watch the hub cluster from an independent vantage point (an external synthetic check or a cross-account canary).

6. Best-practice checklist

Day 1 (seed / first GitOps setup)

One Argo CD, on a cluster that isn’t prod. SSO login, read-only default RBAC.
Every Application has the resources-finalizer.argocd.argoproj.io finalizer from creation.
automated.prune: true, a bounded retry (5 attempts, capped backoff), selfHeal: true in prod.
Image tags are git SHAs; ECR tag immutability is on for every repository.
Promotions are commits: a bot PR per environment that edits only image.tag.
Prod promotion requires an explicit, fail-closed human approval, and every bypass path is documented as break-glass.
The CD workflow reverts the tag in git when a rollout fails.
Deploy-wait logic succeeds on “target image live and Healthy”, not on “Synced”.
Workflow inputs are passed through env:, never interpolated into run:.

First year (Series A)

AppProjects: platform vs <env>-workloads vs key-holding services. No app in default. clusterResourceWhitelist: [] for workload projects.
The library chart encodes sync waves: secrets −1, config/Service 0, workload 1, HPA and other dependents 2.
A real gate on the two or three services where a bad deploy costs money: E2E Job analysis, plus SLI analysis scoped to the canary hash, with empty results counting as failure.
Services with no analysis rendered as plain Deployments. No steps: [] theater.
Helm-diff CI renders every changed app and the merge-to-apply transition state; it fails on a missing opt-in values file.
A prod-scoped OutOfSync for 30m alert, plus a Rollout Degraded/Paused alert, each with a runbook link.
A written list of multi-merge procedures (values file moves, alert rule after its scrape job, port moves).
An inventory of Terraform-managed helm_releases, and how to verify each one deployed.
Code-owner review required for the workflow directory, approval scripts and AppProjects.

At scale

ApplicationSets (cluster and git generators) for the fleet; Terraform only for the hub, projects and cluster registration.
Traffic-routed canaries (ALB/Traefik/mesh) for high-RPS services; replica-based canaries retired.
Digest pinning plus signature verification at admission (cosign/Kyverno or similar), so prod runs only images CI built and signed.
A versioned library chart, promoted per environment.
Sync windows for high-risk periods (token launches, scheduled protocol upgrades).
DORA metrics derived from the git history and the CD workflow’s deployment records [25].
An evaluation of the EKS managed Argo CD capability [21] vs continued self-hosting.

7. Maturity roadmap

Stage	Argo CD	Progressive delivery	Promotion and approval	Observability of delivery
0: kubectl/Helm from CI	None	Rolling updates	CI deploys on merge	CI logs
1: Basic GitOps	Hub Argo CD, app-of-apps or a Terraform module, `default` project	Deployments; Rollouts maybe installed but empty	Bot PRs edit tags; manual prod dispatch	Argo CD UI; Slack `on-sync-failed`
2: Safe defaults	Bounded retry, prod self-heal, finalizers, sync waves in the library	One or two services with E2E-gated blue/green	Fail-closed prod approval issue; auto git-revert on failure; diff-based promotion classification	`OutOfSync 30m` prod alert; deploy annotations on dashboards
3: Real gating	AppProjects per tier; transition-state diff CI; multi-merge procedures documented	SLI analysis on canaries for revenue-critical services; empty canaries replaced by Deployments	Code-owner-protected workflows; supersede logic; hotfix path that clears itself	Rollout phase alerts; DORA change-fail rate and MTTR tracked
4: Fleet scale	ApplicationSets; possibly managed Argo CD; library chart versioned	Traffic-routed canaries; automated analysis everywhere it is meaningful	Signed, digest-pinned images verified at admission (SLSA-style provenance [26])	Per-service delivery SLOs; cross-account watchdog for the hub

The usual mistake is skipping from stage 1 to stage 4’s tooling without stage 2’s semantics: installing Rollouts, an image updater and ApplicationSets while Applications still get stuck on a failed SHA and nobody is alerted. Do stage 2 first. It’s mostly configuration, and it closes the failure modes that actually occur.

The Google SRE Book’s release engineering chapter [18] makes the underlying point: hermetic, reproducible, policy-enforced releases come from a pipeline you trust, not from heroics. AWS Prescriptive Guidance’s GitOps tool comparison [20] is useful if you are still choosing between Argo CD and Flux. The deciding factors for most small teams are multi-cluster management from one UI and application-level health, both of which favor Argo CD.

8. References

OpenGitOps, GitOps Principles v1.0.0. https://opengitops.dev/
Argo CD documentation, Automated Sync Policy (automated sync semantics, self-heal, retry, retry refresh). https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/
Argo CD documentation, Sync Phases and Waves. https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/
Argo CD documentation, Multiple Sources for an Application. https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/
Argo CD documentation, Projects (the default project’s permissive spec). https://argo-cd.readthedocs.io/en/stable/user-guide/projects/
Argo CD documentation, Cluster Bootstrapping (app of apps). https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/
Argo CD documentation, ApplicationSet Controller. https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/
Argo CD documentation, Diffing Customization. https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/
Argo CD documentation, Sync Options (Prune=false, RespectIgnoreDifferences). https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/
Argo CD documentation, Metrics (argocd_app_info). https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/
Argo CD documentation, Best Practices. https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/
Argo Rollouts documentation, Canary Deployment Strategy. https://argoproj.github.io/argo-rollouts/features/canary/
Argo Rollouts documentation, BlueGreen Deployment Strategy. https://argoproj.github.io/argo-rollouts/features/bluegreen/
Argo Rollouts documentation, Analysis & Progressive Delivery. https://argoproj.github.io/argo-rollouts/features/analysis/
Argo Rollouts documentation, Prometheus analysis provider. https://argoproj.github.io/argo-rollouts/analysis/prometheus/
Argo Rollouts documentation, Job analysis provider. https://argoproj.github.io/argo-rollouts/analysis/job/
Google, The Site Reliability Workbook, ch. “Canarying Releases”. https://sre.google/workbook/canarying-releases/
Google, Site Reliability Engineering, ch. 8 “Release Engineering”. https://sre.google/sre-book/release-engineering/
AWS Well-Architected Framework, Reliability Pillar, REL08-BP05 Deploy changes with automation. https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_tracking_change_management_automated_changemgmt.html
AWS Prescriptive Guidance, Choosing the right GitOps tool for your Amazon EKS cluster. https://docs.aws.amazon.com/prescriptive-guidance/latest/eks-gitops-tools/introduction.html
Amazon EKS User Guide, Continuous Deployment with Argo CD (EKS Capabilities). https://docs.aws.amazon.com/eks/latest/userguide/argocd.html
Amazon EKS, Best Practices Guide. https://docs.aws.amazon.com/eks/latest/best-practices/introduction.html
Martin Fowler, BlueGreenDeployment; Danilo Sato, CanaryRelease (martinfowler.com). https://martinfowler.com/bliki/BlueGreenDeployment.html and https://martinfowler.com/bliki/CanaryRelease.html
Kubernetes documentation, Horizontal Pod Autoscaling. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
DORA, DORA’s software delivery metrics: the four keys. https://dora.dev/guides/dora-metrics-four-keys/
SLSA, Supply-chain Levels for Software Artifacts, v1.0 specification. https://slsa.dev/spec/v1.0/
GitHub Docs, Security hardening for GitHub Actions (script injection, untrusted input). https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions
HashiCorp Terraform Kubernetes provider, kubernetes_manifest resource (field_manager, computed_fields). https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest