❯devopsindex

ArgoCD

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes. Git is the single source of truth; ArgoCD continuously syncs the cluster to match it.

Most sections below end with a quick knowledge check — track how many you've cleared as you go:

0/0 checks

GitOps Model

graph LR
    classDef blue fill:#3498db,stroke:#2980b9,color:#fff
    classDef green fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef red fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef orange fill:#e67e22,stroke:#d35400,color:#fff
    classDef purple fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef teal fill:#1abc9c,stroke:#16a085,color:#fff
    classDef dark fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef yellow fill:#f39c12,stroke:#d68910,color:#000
    classDef k8s fill:#326ce5,stroke:#254ea8,color:#fff
    classDef aws fill:#ff9900,stroke:#cc7a00,color:#000
    DEV["Developer pushes K8s manifest to git"]:::green --> GIT["Git repo (manifests, Helm charts, Kustomize)"]:::green
    GIT --> ARGO["ArgoCD controller watches git repo"]:::green
    ARGO --> DIFF["Computes diff: git state vs cluster state"]:::green
    DIFF -->|"out of sync"| SYNC["Syncs cluster to match git (kubectl apply equivalent)"]:::green
    SYNC --> CLUSTER["Kubernetes Cluster"]:::k8s
    CLUSTER -->|"health checks"| ARGO

That diagram is a loop, not a one-shot pipeline. Step through what actually happens, one stage at a time:

1. Developer pushes. A K8s manifest, Helm chart, or Kustomize change lands in the git repo — the single source of truth.
2. ArgoCD's controller watches. It's continuously watching the git repo, not waiting to be triggered by a pipeline.
3. Computes a diff. Git state vs live cluster state, resource by resource.
4. Syncs if out of sync. Applies whatever's needed to make the cluster match git — the kubectl apply equivalent.
5. Health checks, then loop. ArgoCD checks the cluster's health and goes right back to watching — this never stops running.
The pipeline itself has cluster credentials and applies changes directly to the cluster.
An agent running inside the cluster pulls changes from git — no external system needs cluster credentials at all.

Why GitOps wins for Kubernetes:

  • Cluster credentials never leave the cluster — no secrets in CI pipelines
  • Full audit trail in git: who changed what, when, and why (via PR description)
  • Rollback = git revert + ArgoCD auto-syncs back
  • Multi-cluster: one ArgoCD instance can manage many clusters

In ArgoCD's pull model, which side actually holds the cluster credentials — the CI pipeline or the in-cluster agent?

Application CRD

The Application is ArgoCD's core resource. It defines what to deploy and where.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  # Where to deploy
  destination:
    server: https://kubernetes.default.svc   # in-cluster
    namespace: production

  # Source: where the K8s manifests live
  source:
    repoURL: https://github.com/my-org/my-app-manifests
    targetRevision: main     # branch, tag, or commit SHA
    path: k8s/production     # directory within repo

  # Sync policy
  syncPolicy:
    automated:
      prune: true      # delete resources removed from git
      selfHeal: true   # revert manual cluster changes back to git state
    syncOptions:
      - CreateNamespace=true

  # Health check — what does "healthy" mean for this app?
  # ArgoCD has built-in health checks for Deployment, StatefulSet, Ingress, etc.

Without automated sync: ArgoCD detects drift but waits for a human to click "Sync" or run argocd app sync my-app. Manual sync = change management gate before applying to production.

You create an Application without a syncPolicy.automated block. A new commit lands in git — does ArgoCD apply it to the cluster automatically?

Sync Policies

graph TD
    classDef blue fill:#3498db,stroke:#2980b9,color:#fff
    classDef green fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef red fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef orange fill:#e67e22,stroke:#d35400,color:#fff
    classDef purple fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef teal fill:#1abc9c,stroke:#16a085,color:#fff
    classDef dark fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef yellow fill:#f39c12,stroke:#d68910,color:#000
    classDef k8s fill:#326ce5,stroke:#254ea8,color:#fff
    classDef aws fill:#ff9900,stroke:#cc7a00,color:#000
    subgraph ManualSync["Manual Sync (default)"]
        DETECT["ArgoCD detects OutOfSync"]:::green --> WAIT["Waits for human approval"]:::orange
        WAIT --> SYNC_BTN["Human clicks Sync / argocd app sync"]:::green
        SYNC_BTN --> APPLY["Apply changes to cluster"]:::blue
    end

    subgraph AutoSync["Automated Sync"]
        A_DETECT["ArgoCD detects OutOfSync"]:::green --> AUTO_APPLY["Immediately applies changes"]:::blue
        AUTO_APPLY --> HEALTH["Health assessment"]:::yellow
        HEALTH -->|"Degraded"| ROLLBACK["Auto-rollback to previous state"]:::blue
    end

    subgraph SelfHeal["Self-Heal (automated + selfHeal: true)"]
        KUBECTL["kubectl edit deployment (manual change)"]:::green --> DRIFT2["ArgoCD detects drift within 3min"]:::red
        DRIFT2 --> REVERT["Reverts manual change to match git"]:::green
    end
Policy Use case
Manual sync Production — require human approval for every deployment
Auto sync without self-heal Staging — auto-deploy new commits, allow temporary manual patches
Auto sync + self-heal Dev/ephemeral — full automation, git is absolute truth
Auto sync + prune Any — clean up resources deleted from git

Auto sync is enabled but selfHeal is not. Someone runs kubectl edit deployment straight against the cluster. Does ArgoCD revert it?

Sync Waves and Hooks

Control the order of resource application within a single sync:

# Database migration job — must complete BEFORE app deployment
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync        # runs before the main sync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded  # clean up after
    argocd.argoproj.io/sync-wave: "-1"      # wave order (lower = earlier)
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: my-app:v2
          command: ["/app/server", "migrate"]
---
# Main deployment — runs after migration completes
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # default wave

Hook types:

Runs before sync — migrations, backups.
Runs during sync, alongside regular resources.
Runs after all resources are healthy — smoke tests, notifications.
Runs if sync fails — alert, rollback trigger.

Put together, one sync's hooks and waves execute in this order:

1. Lowest sync-wave first. A PreSync hook at wave -1 (like the db-migrate Job above) runs before anything else.
2. Hook cleanup. With hook-delete-policy: HookSucceeded, ArgoCD deletes the hook resource once it succeeds — it did its job.
3. Next wave applies. Wave 0 (the default) — the main Deployment in this example — gets applied next.
4. Sync hooks, if any. Run alongside the regular resources during the sync itself.
5. PostSync or SyncFail. Once every resource is applied and healthy, PostSync hooks fire (smoke tests, notifications). If anything failed along the way, SyncFail hooks fire instead (alerting, rollback trigger).

A hook has argocd.argoproj.io/hook: PreSync and sync-wave: "-1"; the main Deployment sits at the default wave 0. Which runs first, and what happens to the hook resource once it succeeds (given hook-delete-policy: HookSucceeded)?

App of Apps Pattern

Manage multiple applications from a single ArgoCD Application. The root app deploys other Application CRDs.

graph TD
    classDef blue fill:#3498db,stroke:#2980b9,color:#fff
    classDef green fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef red fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef orange fill:#e67e22,stroke:#d35400,color:#fff
    classDef purple fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef teal fill:#1abc9c,stroke:#16a085,color:#fff
    classDef dark fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef yellow fill:#f39c12,stroke:#d68910,color:#000
    classDef k8s fill:#326ce5,stroke:#254ea8,color:#fff
    classDef aws fill:#ff9900,stroke:#cc7a00,color:#000
    ROOT["ArgoCD Application: apps-root points to: clusters/prod/apps/"]:::blue --> APP1["Application CRD: payments-service"]:::teal
    ROOT --> APP2["Application CRD: api-gateway"]:::orange
    ROOT --> APP3["Application CRD: worker-service"]:::teal
    ROOT --> APP4["Application CRD: monitoring-stack"]:::blue

    APP1 --> DEPLOY1["Deploys: payments deployment + service + ingress"]:::green
    APP2 --> DEPLOY2["Deploys: api-gateway resources"]:::green
    APP4 --> DEPLOY4["Deploys: Prometheus + Grafana + Alertmanager"]:::green

clusters/prod/apps/
├── apps-root.yaml          # root Application
├── payments-service.yaml   # Application pointing to payments manifests
├── api-gateway.yaml
└── monitoring-stack.yaml   # Application pointing to monitoring Helm chart

Why: New applications are added by adding a new Application YAML to git — ArgoCD picks it up automatically. No manual ArgoCD configuration required.

You need to add a fifth application under the App of Apps setup above. What do you have to configure inside ArgoCD itself?

Multi-Cluster Management

One ArgoCD instance can manage many clusters:

# Register external cluster
argocd cluster add my-prod-cluster --name production

# Application targeting external cluster
spec:
  destination:
    server: https://my-prod-cluster-api-endpoint:6443
    namespace: payments

graph TD
    classDef blue fill:#3498db,stroke:#2980b9,color:#fff
    classDef green fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef red fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef orange fill:#e67e22,stroke:#d35400,color:#fff
    classDef purple fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef teal fill:#1abc9c,stroke:#16a085,color:#fff
    classDef dark fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef yellow fill:#f39c12,stroke:#d68910,color:#000
    classDef k8s fill:#326ce5,stroke:#254ea8,color:#fff
    classDef aws fill:#ff9900,stroke:#cc7a00,color:#000
    ARGO_CTRL["ArgoCD Controller in management cluster"]:::purple --> CLUSTER_PROD["Production cluster us-east-1"]:::blue
    ARGO_CTRL --> CLUSTER_STG["Staging cluster us-east-1"]:::blue
    ARGO_CTRL --> CLUSTER_EU["EU cluster eu-west-1"]:::blue

    GIT["Git: manifests per cluster clusters/prod/ clusters/staging/ clusters/eu/"]:::green --> ARGO_CTRL

ApplicationSet — generate Applications automatically for each cluster from a template:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app-all-clusters
spec:
  generators:
    - clusters: {}   # generates one Application per registered cluster
  template:
    spec:
      source:
        repoURL: https://github.com/my-org/manifests
        path: "clusters/{{name}}"   # {{name}} = cluster name
      destination:
        server: "{{server}}"        # cluster API endpoint
        namespace: my-app

You already have an ApplicationSet using the clusters generator with no selector. You register a brand-new cluster with argocd cluster add. Do you need to write it a new Application manifest?

Rollback

# View sync history
argocd app history my-app

# Rollback to previous revision
argocd app rollback my-app <revision-id>

# Or: git revert the commit, ArgoCD auto-syncs back
git revert HEAD
git push origin main
# ArgoCD detects the new commit and syncs (which undoes the bad deploy)

Git revert is the preferred rollback — it's auditable, goes through the same review process, and keeps a clean history. ArgoCD's built-in rollback is useful for emergency hotfixes.

Both argocd app rollback and git revert + push undo a bad deploy. Which one is preferred, and why?

Image Updater

ArgoCD Image Updater automatically updates image tags in git when new images are pushed:

# Annotation on Application
annotations:
  argocd-image-updater.argoproj.io/image-list: myapp=123456.dkr.ecr.us-east-1.amazonaws.com/my-app
  argocd-image-updater.argoproj.io/myapp.update-strategy: semver   # or digest, latest, name
  argocd-image-updater.argoproj.io/write-back-method: git          # commit new tag to git

Flow:

  1. CI builds and pushes my-app:v1.2.3 to ECR
  2. Image Updater polls ECR, detects new tag
  3. Updates the image tag in the git repo (via commit)
  4. ArgoCD detects the git change, syncs the cluster
  5. Full GitOps: no CI pipeline needs cluster credentials
1. CI builds. Pushes my-app:v1.2.3 to ECR — that's the last thing CI does.
2. Image Updater polls. It watches ECR directly and notices the new tag on its own.
3. Commits to git. It updates the image tag in the git repo itself — not by touching the cluster.
4. ArgoCD syncs. It sees that git change exactly like any other commit, and syncs the cluster to match.
5. Full GitOps, end to end. No CI pipeline ever needed cluster credentials — git stayed the single source of truth the whole way through.

With write-back-method: git configured, does the CI pipeline that builds my-app:v1.2.3 need cluster credentials to get that image deployed?

Health Checks and Notifications

ArgoCD has built-in health assessments for standard K8s resources. Custom health checks via Lua scripts:

# config in argocd-cm ConfigMap
resource.customizations.health.argoproj.io_Rollout: |
  hs = {}
  if obj.status ~= nil then
    if obj.status.phase == "Healthy" then
      hs.status = "Healthy"
    elseif obj.status.phase == "Degraded" then
      hs.status = "Degraded"
      hs.message = obj.status.message
    end
  end
  return hs

Notifications — alert on sync events via Slack, PagerDuty, email:

# argocd-notifications-cm
triggers:
  - name: on-sync-failed
    condition: app.status.sync.status == 'Unknown'
    template: app-sync-failed

templates:
  - name: app-sync-failed
    message: |
      Application {{.app.metadata.name}} sync failed.
      Error: {{.app.status.conditions[0].message}}

Multi-Cluster Deployments

Where does ArgoCD live?

ArgoCD is installed in one cluster — the "management cluster" (or hub cluster). It then connects to N target clusters and deploys into them. ArgoCD itself never needs to run in every cluster.

graph TD
    classDef blue fill:#3498db,stroke:#2980b9,color:#fff
    classDef green fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef orange fill:#e67e22,stroke:#d35400,color:#fff
    classDef k8s fill:#326ce5,stroke:#254ea8,color:#fff

    GIT["Git repo: manifests per cluster"]:::green

    subgraph MGMT["Management Cluster (ArgoCD lives here)"]
        ARGO["ArgoCD<br/>argocd-server<br/>argocd-application-controller"]:::blue
    end

    subgraph C1["Cluster 1: production-us"]
        APP1["Deployed apps"]:::k8s
    end
    subgraph C2["Cluster 2: production-eu"]
        APP2["Deployed apps"]:::k8s
    end
    subgraph C3["Cluster 3: staging"]
        APP3["Deployed apps"]:::k8s
    end

    GIT -->|watches| ARGO
    ARGO -->|kubectl API via kubeconfig/ServiceAccount| C1
    ARGO -->|kubectl API| C2
    ARGO -->|kubectl API| C3

Connectivity: ArgoCD connects to target clusters using a ServiceAccount token. It uses the Kubernetes API — needs network access from the management cluster to target cluster API servers (typically via private endpoint, VPN, or VPC peering).

Step 1: Register target clusters

# Login to ArgoCD CLI
argocd login <argocd-server-host>

# Add each target cluster (uses your current kubeconfig context)
argocd cluster add production-us-context --name production-us
argocd cluster add production-eu-context --name production-eu
argocd cluster add staging-context       --name staging

# Verify
argocd cluster list
# SERVER                          NAME             STATUS
# https://prod-us.eks.amazonaws.com   production-us    Successful
# https://prod-eu.eks.amazonaws.com   production-eu    Successful
# https://staging.eks.amazonaws.com   staging          Successful

This creates a Secret in the argocd namespace with the cluster credentials.

Step 2: One Application per cluster (manual)

# app-production-us.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp-production-us
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/k8s-manifests
    targetRevision: main
    path: apps/myapp/overlays/production-us   # kustomize overlay per cluster
  destination:
    server: https://prod-us.eks.amazonaws.com  # target cluster API
    namespace: myapp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Repeat for each cluster. This works but doesn't scale — 3 clusters × 10 apps = 30 Application manifests.

Step 3: ApplicationSet — the scalable way

ApplicationSet is an ArgoCD controller that generates Application objects automatically from a template + generator. One manifest, N applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: myapp
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: production-us
        url: https://prod-us.eks.amazonaws.com
        env: production
      - cluster: production-eu
        url: https://prod-eu.eks.amazonaws.com
        env: production
      - cluster: staging
        url: https://staging.eks.amazonaws.com
        env: staging
  template:
    metadata:
      name: "myapp-{{cluster}}"          # generates: myapp-production-us, myapp-production-eu, myapp-staging
    spec:
      project: default
      source:
        repoURL: https://github.com/my-org/k8s-manifests
        targetRevision: main
        path: "apps/myapp/overlays/{{cluster}}"   # per-cluster kustomize overlay
      destination:
        server: "{{url}}"
        namespace: myapp
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

This generates 3 Application objects automatically. Add a 4th cluster → add one list element.

Git repo structure for multi-cluster

k8s-manifests/
├── apps/
│   └── myapp/
│       ├── base/                    # shared manifests (Deployment, Service)
│       │   ├── deployment.yaml
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── production-us/       # cluster-specific patches
│           │   ├── kustomization.yaml
│           │   └── patch-replicas.yaml   # replicas: 10
│           ├── production-eu/
│           │   ├── kustomization.yaml
│           │   └── patch-replicas.yaml   # replicas: 5
│           └── staging/
│               ├── kustomization.yaml
│               └── patch-replicas.yaml   # replicas: 2

Progressive delivery across clusters

Deploy to staging first, promote to prod after health check:

# Use sync waves: staging syncs first, prod only after staging is healthy
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"    # staging: wave 1 (first)
    # prod clusters: wave 2 (after wave 1 healthy)

Or use ArgoCD's Rollout integration with Argo Rollouts for canary/blue-green per cluster.

Hub-spoke vs. standalone ArgoCD per cluster

Hub (one ArgoCD, N clusters) Standalone (ArgoCD per cluster)
Ops overhead One ArgoCD to maintain N ArgoCD instances
Blast radius ArgoCD cluster down = no deploys to any cluster One cluster's ArgoCD down = only that cluster affected
Network Needs API access to all target clusters Only needs access to own cluster
Use case Small/medium orgs, central platform team Large orgs, strict isolation, air-gapped clusters

Projects and RBAC

ArgoCD Projects group Applications and enforce what can be deployed where. RBAC controls who can do what.

AppProject — boundaries for a team

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: "Payments team — production and staging"

  # Which git repos this project can source from
  sourceRepos:
    - https://github.com/my-org/payments-manifests
    - https://github.com/my-org/shared-charts

  # Which clusters + namespaces Applications in this project can deploy to
  destinations:
    - server: https://kubernetes.default.svc
      namespace: payments-prod
    - server: https://kubernetes.default.svc
      namespace: payments-staging

  # Block deploying cluster-scoped resources (RBAC, CRDs, etc.)
  clusterResourceWhitelist: []          # empty = no cluster-scoped resources allowed

  # Allow only these namespaced resource types
  namespaceResourceWhitelist:
    - group: apps
      kind: Deployment
    - group: ""
      kind: Service
    - group: ""
      kind: ConfigMap
    - group: autoscaling
      kind: HorizontalPodAutoscaler

  # Sync windows — prevent deploys outside business hours in prod
  syncWindows:
    - kind: deny
      schedule: "0 22 * * *"    # deny at 10pm every day
      duration: 8h               # for 8 hours (10pm–6am)
      applications: ["*"]
      namespaces: ["payments-prod"]
    - kind: allow
      schedule: "0 9 * * 1-5"   # allow weekdays 9am
      duration: 9h
      applications: ["*"]
      namespaces: ["payments-prod"]

RBAC — who can do what

ArgoCD RBAC is configured in the argocd-rbac-cm ConfigMap. Roles are defined with p (policy) lines, then bound to users/groups.

# kubectl edit configmap argocd-rbac-cm -n argocd
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly   # default role for authenticated users

  policy.csv: |
    # Format: p, <role>, <resource>, <action>, <object>, allow|deny

    # Platform team — full access to everything
    p, role:platform-admin, *, *, *, allow

    # Payments team — can sync/manage apps in team-payments project only
    p, role:payments-dev, applications, get,      team-payments/*, allow
    p, role:payments-dev, applications, sync,     team-payments/*, allow
    p, role:payments-dev, applications, action/*, team-payments/*, allow
    p, role:payments-dev, logs,         get,      team-payments/*, allow
    p, role:payments-dev, exec,         create,   team-payments/*, deny   # no exec into pods

    # Read-only role for external auditors
    p, role:auditor, applications, get, */*, allow
    p, role:auditor, clusters,     get, *,   allow

    # Bind GitHub SSO groups to roles
    g, my-org:platform-team, role:platform-admin
    g, my-org:payments,      role:payments-dev
    g, my-org:security-audit, role:auditor

Resource types for RBAC:

Resource Actions
applications get, create, update, delete, sync, override, action/*
applicationsets get, create, update, delete
clusters get, create, update, delete
repositories get, create, update, delete
logs get
exec create

SSO Integration (Dex + GitHub)

# argocd-cm ConfigMap
data:
  url: https://argocd.my-company.com

  dex.config: |
    connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $dex.github.clientID        # from argocd-secret
        clientSecret: $dex.github.clientSecret
        orgs:
        - name: my-org
          teams:
          - platform-team
          - payments
          - security-audit

# Verify RBAC — test what a user can do
argocd admin settings rbac can role:payments-dev sync applications team-payments/my-app
# → true or false

# List effective permissions for a user
argocd admin settings rbac can --user john@my-org.com sync applications team-payments/my-app

Project best practices

One AppProject per team (not per application)
  → Team owns all their apps under one project
  → Limits blast radius: team can't accidentally deploy to another team's namespace

Always set clusterResourceWhitelist: []
  → Prevents teams from creating ClusterRoles, CRDs, etc.
  → Platform team's project is the only one with cluster resource access

Use syncWindows on production projects
  → Prevents deploys during peak traffic hours
  → Exception: allow override for emergency hotfixes (manual sync with --override)

Separate projects for prod vs non-prod
  → Different sync policies (manual for prod, auto for staging)
  → Different RBAC (only platform team can sync prod)

Sync Failure Runbook

Step 1 — read the sync error

# CLI
argocd app get payments --show-operation

# Or describe the Application CRD
kubectl describe application payments -n argocd
# Look at: Status.Conditions, Status.OperationState.Message

# Common error messages and meanings:
# "rpc error: code = Unknown desc = Forbidden"
#   → ArgoCD SA lacks permission to create/update this resource
# "existing object ... is not managed by argocd"
#   → Resource exists but wasn't created by ArgoCD (no owner annotation)
#     Fix: argocd app sync payments --force  OR  kubectl annotate ...
# "one or more objects failed to apply"
#   → Schema validation error; look for "must be" or "field is required"
# "ComparisonError: failed to get local objects"
#   → Cannot render Helm/Kustomize templates; syntax error in values

Step 2 — sync status taxonomy

Status Meaning Action
Synced Live == desired None
OutOfSync Live ≠ desired Review diff, sync
Unknown Can't compare (render failed) Fix template/values
SyncFailed Apply failed Read operation message
Missing Resource not in cluster yet Sync will create it

Step 3 — common failure scenarios

Webhook timeout (sync job runs but never finishes):

# ArgoCD sync has a default timeout of 1 hour
# For large apps: extend it
argocd app set payments --sync-option Timeout=3600

# Or in Application spec:
spec:
  syncPolicy:
    syncOptions:
      - Timeout=3600

Resource exists but ArgoCD didn't create it (out-of-band resource):

# ArgoCD refuses to overwrite resources it doesn't own
# Option 1: adopt the resource
kubectl annotate deployment payments \
  argocd.argoproj.io/managed-by=argocd -n payments

# Option 2: force sync (replaces regardless of ownership)
argocd app sync payments --force

# Option 3: configure app to adopt all orphaned resources
spec:
  syncPolicy:
    syncOptions:
      - Replace=true   # uses kubectl replace instead of apply

CRD missing — app requires CRD that doesn't exist yet:

# Use sync waves: install CRDs first (wave -1), then CRD-dependent resources (wave 0+)
# On the CRD manifest:
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

# On resources that depend on the CRD:
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"

Hook job failing:

# Jobs with argocd.argoproj.io/hook: PreSync that fail block the entire sync
kubectl get jobs -n payments
kubectl logs job/<hook-job-name> -n payments

# Delete failed hook to unblock sync
kubectl delete job <hook-job-name> -n payments
argocd app sync payments

ApplicationSet — Generators

ApplicationSet automates creating many ArgoCD Applications from a single template. Instead of one Application YAML per service/cluster, you write one ApplicationSet.

Git generator — one app per directory

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: microservices
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/myorg/config-repo
        revision: main
        directories:
          - path: services/*     # one app per directory under services/
          # exclude: services/experimental  ← can exclude patterns
  template:
    metadata:
      name: '{{path.basename}}'  # app name = directory name
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/config-repo
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Matrix generator — every service × every cluster

generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/myorg/config-repo
            revision: main
            files:
              - path: services/*/config.json   # produces {service: "payments", ...}
        - list:
            elements:
              - cluster: staging
                url: https://staging.k8s.example.com
              - cluster: production
                url: https://prod.k8s.example.com
template:
  metadata:
    name: '{{service}}-{{cluster}}'
  spec:
    destination:
      server: '{{url}}'
      namespace: '{{service}}'
    source:
      path: 'services/{{service}}/{{cluster}}'

Cluster generator — deploy to all registered clusters

generators:
  - clusters:
      selector:
        matchLabels:
          environment: production   # only prod clusters
      # All clusters if no selector
template:
  metadata:
    name: 'monitoring-{{name}}'    # {{name}} = cluster name in ArgoCD
  spec:
    destination:
      server: '{{server}}'         # {{server}} = cluster API URL
      namespace: monitoring
    source:
      repoURL: https://charts.myorg.com
      chart: monitoring-stack
      targetRevision: 2.0.0

Pull Request generator — ephemeral preview environments

generators:
  - pullRequest:
      github:
        owner: myorg
        repo: config-repo
        tokenRef:
          secretName: github-token
          key: token
        labels:
          - preview           # only PRs with this label
template:
  metadata:
    name: 'preview-{{number}}'    # {{number}} = PR number
  spec:
    destination:
      namespace: 'preview-{{number}}'
    source:
      helm:
        parameters:
          - name: image.tag
            value: 'pr-{{number}}'
    syncPolicy:
      syncOptions:
        - CreateNamespace=true

Health Check Customization

ArgoCD uses built-in health checks for core K8s resources. You can override them with Lua scripts.

Custom resource health check

# In argocd-cm ConfigMap:
data:
  resource.customizations.health.myorg.io_MyDatabase: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.phase == "Running" then
        hs.status = "Healthy"
      elseif obj.status.phase == "Failed" then
        hs.status = "Degraded"
        hs.message = obj.status.message
      end
    end
    return hs

Override Deployment health to check custom condition

resource.customizations.health.apps_Deployment: |
  hs = {}
  if obj.status ~= nil then
    if obj.status.conditions ~= nil then
      for i, condition in ipairs(obj.status.conditions) do
        if condition.type == "MyCustomReady" and condition.status == "False" then
          hs.status = "Degraded"
          hs.message = condition.message
          return hs
        end
      end
    end
  end
  -- Fall through to default Deployment health check logic
  hs.status = "Healthy"
  return hs

Ignore differences — stop OutOfSync noise

Some controllers mutate resources after ArgoCD applies them (e.g., adding generation, resourceVersion, or controller-managed fields). Configure ArgoCD to ignore these fields:

# In Application spec:
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas          # HPA manages replicas; ignore ArgoCD's desired count
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP         # assigned by K8s, not in git
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle   # cert-manager injects this

# Global ignore (argocd-cm):
data:
  resource.customizations.ignoreDifferences.apps_Deployment: |
    jqPathExpressions:
      - .spec.replicas
← Previous
Jenkins Master-Agent on AWS ECS
Next →
GitOps Secrets Management