Enterprise GitOps Pipelines with ArgoCD & Helm
This sets up a multi-environment GitOps pipeline with ArgoCD and Helm, using an app-of-apps structure and per-environment value overlays, with automated sync policies that continuously reconcile the cluster against Git and self-heal drift instead of just detecting it.
The app-of-apps pattern
At any scale beyond a handful of services, you don't want one Application per microservice; you want a single root Application that itself manages every other Application. One Git commit to the apps repo is enough to onboard, decommission, or re-point an entire environment.
# root-app.yaml, the single object a cluster admin ever touches manually
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-apps
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/acme/gitops-apps.git
targetRevision: main
path: apps/production
directory:
recurse: true
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: trueEverything under apps/production/ is a plain ArgoCD Application manifest, one per service or one per logical group. The root app watches that directory; add a file, get a new managed app, delete a file, get a clean teardown.
Helm chart structure and per-environment overlays
Under the app-of-apps layer, each service is a Helm chart with a values file per environment rather than per-environment branches or forked repos. Branching your manifests is how you end up with staging and production silently diverging in ways nobody notices until the postmortem.
charts/payments-api/
├── Chart.yaml
├── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── hpa.yaml
├── values.yaml # shared defaults
├── values-staging.yaml # overlay
└── values-production.yaml # overlay# values-production.yaml
replicaCount: 6
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
autoscaling:
enabled: true
minReplicas: 6
maxReplicas: 24
env:
LOG_LEVEL: warn
FEATURE_FLAGS_SOURCE: launchdarkly-prodvalues.yaml never sets replica counts or resource limits directly; every environment-sensitive field lives in the overlay, and the diff between staging and production is a diff between two YAML files, not two chart versions.
Wiring the Application CRD to Helm with automated sync
Each child Application points at the chart and the environment overlay, and turns on automated sync with prune and selfHeal.
# apps/production/payments-api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: payments-api
namespace: argocd
spec:
project: production
source:
repoURL: https://github.com/acme/gitops-apps.git
targetRevision: main
path: charts/payments-api
helm:
valueFiles:
- values.yaml
- values-production.yaml
destination:
server: https://kubernetes.default.svc
namespace: payments
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
retry:
limit: 5
backoff:
duration: 10s
factor: 2
maxDuration: 3mprune: true means resources removed from the chart get removed from the cluster on the next sync. Without it, deleted templates leave orphaned objects behind indefinitely. selfHeal: true reverts any manual change to a live object back to what's in Git, automatically, on the next reconciliation pass (by default every three minutes, or immediately if you're watching the resource).
The fix here is procedural, not technical: your incident runbook needs an explicit step to either commit the patch immediately or pause sync (argocd app set <app> --sync-policy none) before anyone touches a live resource by hand.
Sync waves and hooks for ordering
Helm's template rendering has no concept of "apply this before that" across separate charts. Plenty of real deployments need it anyway: CRDs before the controllers that watch them, or a migration Job that has to finish before the Deployment depending on the new schema comes up. ArgoCD's sync waves solve this with a plain annotation.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: postgresqls.acme.io
annotations:
argocd.argoproj.io/sync-wave: "-1"
---
apiVersion: batch/v1
kind: Job
metadata:
name: payments-db-migrate
annotations:
argocd.argoproj.io/sync-wave: "0"
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
annotations:
argocd.argoproj.io/sync-wave: "1"Lower waves sync first, and ArgoCD waits for each wave's resources to be Healthy before moving to the next. Combined with PreSync hooks for one-shot jobs like migrations, this replaces the fragile "run this script before that Helm release" step.
Drift detection and the manual override workflow
Even with selfHeal on, you need a documented path for legitimate manual intervention. When an incident genuinely requires a live patch faster than a PR-and-merge cycle allows, the workflow is: pause auto-sync on the specific Application, make the change, then either revert it or fast-follow it into Git within the same incident window. Re-enable auto-sync only after the app is confirmed in sync.
# temporarily disable automated sync during the incident
spec:
syncPolicy: {} # remove the `automated` block; manual sync onlyOutside of an active incident, drift shows up in the ArgoCD UI and API as OutOfSync. It's worth alerting on directly: more than a few minutes of it on a production app, with no incident in progress, means someone bypassed the process.
Set revisionHistoryLimit on each Application deliberately (ArgoCD keeps 10 historical manifests by default). On a chart with large CRDs or many managed resources, that history is what makes argocd app rollback slow during exactly the incident where you need it fast.
Someone will still patch a live resource by hand during a bad incident. What changes is what happens next: either the patch lands in Git and reconciliation confirms it, or selfHeal quietly reverts it and you find out in minutes instead of at the next unrelated deploy.
Want to actually run this in production?
This tutorial covers the concepts and architecture. If you want to implement it in your own infrastructure, or get good enough to own this problem long-term, I offer 1:1 mentoring built around your real environment, not a generic course.
This tutorial
- Core architecture & key concepts
- Illustrative code snippets
- The reasoning behind each decision
1:1 mentoring
- Working sessions on your own environment
- Direct answers to the edge cases you're hitting
- Feedback on your actual implementation
- Ongoing support as you build it out