Promoting Changes from Dev to Staging to Prod with Argo CD
Walk me through how a change moves from dev to staging to prod in an Argo CD setup. How do you stop something from slipping into prod that should not be there?
Walk me through how a change moves from dev to staging to prod in an Argo CD setup. How do you stop something from slipping into prod that should not be there?
Promotion in GitOps is a Git operation, not a kubectl operation. A pull request into the manifests repo updates the prod overlay. The Argo CD controller picks up the new commit and reconciles. The three things to get right: what triggers each environment, how image tags move between environments, and how prod is gated. A flow that works: 1. Developer merges code into the application repo. CI builds an image and pushes it with an immutable tag, the Git SHA or a semver. 2. CI commits the new tag to apps/web/overlays/dev/kustomization.yaml on main of the manifests repo. Argo CD dev Application has auto-sync on and self-heal on. It picks up the change and ships to dev within a couple of minutes. 3. Promotion to staging is a pull request bumping apps/web/overlays/staging/kustomization.yaml to the same tag. Merge after a smoke test or an automated check. Argo CD syncs staging. 4. Promotion to prod is another pull request, this time bumping the prod overlay. Branch protection requires code owners, a passing CI check, and a manual reviewer. After merge, Argo CD syncs prod, or a human clicks sync if prod is set to manual. Non-negotiables I enforce: - Same image artifact through every environment. Do not rebuild between staging and prod. The binary that passed staging is the one that ships. - Immutable tags. No latest, no main. Use the Git SHA or a release version. - Prod sync policy is different from dev. Either manual sync, or auto-sync with sync windows so it can only run during business hours. - selfHeal is fine for dev but think hard about it on prod. It reverts any manual change, which can cause grief during an incident if the on-call engineer needs a quick patch. If you keep selfHeal off on prod, you also need a clear policy that the only way to change prod is through Git. Gotcha that bites teams: auto-sync off does not mean self-heal off. They are separate. If self-heal is on, Argo CD will still revert manual edits, even though it will not pull new commits automatically. Configure both knobs deliberately. Hotfixes: have a documented bypass that still goes through Git. Commit straight to the prod overlay path with senior approval. Never apply manifests with kubectl from a laptop just because the PR flow feels slow.
This question covers the operational side of GitOps end to end. Strong candidates have a clear, opinionated answer with specific knobs they would set. They will mention immutable tags, sync windows, branch protection on the manifests repo, and the difference between auto-sync and self-heal. Weak candidates describe a flow but cannot answer 'what stops a developer from running kubectl apply against prod' or 'what happens when selfHeal is on during an incident.' This is where you find out if they have actually run Argo CD past the demo stage.
Prod Application with manual sync and stricter policy
AppProject with sync windows that limit prod to business hours
Promotion script: bump staging to the tag currently in dev
- Rebuilding the image between staging and prod so a different binary ships than was tested
- Using a floating tag like latest or main on prod, so what gets deployed depends on when sync runs rather than what is in Git
- Enabling auto-sync and self-heal on prod with the same settings as dev, then being surprised when self-heal reverts an on-call hotfix
- A change works in staging and breaks in prod because of a config that differed. How do you stop that next time?
- How do database migrations fit into this flow? They are not idempotent the way manifests are.
- What is your rollback story? A bad change went out, the Argo CD UI is open, what do you do?
- How does this change if you are running progressive delivery with Argo Rollouts in front of the Deployment?
More GitOps interview questions
Also worth your time on this topic
Argo CD Multi-Environment Repository Structure Checklist
How to organize your Git repositories when running Argo CD across dev, staging, and production. Covers folder layout, app-of-apps, ApplicationSets, secrets, RBAC, and promotion flow.
60-90 minutes
Structuring a Git Repo for Argo CD Multi-Environment Deployments
How would you structure a Git repo for Argo CD when you have dev, staging, and prod environments?
junior
GitOps with Argo CD: Structuring Your Repository for Multi-Environment Deployments
A practical guide to laying out your Git repository for Argo CD across dev, staging, and production. See real folder structures, Kustomize and Helm patterns, and the pitfalls that bite teams in production.