CI/CD Debugging Scenarios

Practical debugging playbooks for GitHub Actions, ArgoCD, and Jenkins.

1. GitHub Actions: OIDC Auth to AWS Fails

Symptom: Error assuming role or Token is not valid when using aws-actions/configure-aws-credentials.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["OIDC auth fails:<br/>Error assuming role /<br/>Token is not valid"]:::err --> B
    B{"Audience claim<br/>matches trust policy?"}:::decision
    B -- No --> B1["Set audience to<br/>sts.amazonaws.com"]:::fix

    subgraph TRUST["Trust policy verification"]
        B -- Yes --> C{"Subject condition<br/>in trust policy?"}:::decision
        C -- Mismatch --> C1["Fix repo/branch in<br/>the Condition block"]:::fix
        C -- OK --> D{"OIDC provider<br/>thumbprint valid?"}:::decision
        D -- Stale --> D1["Update thumbprint in<br/>the IAM OIDC provider"]:::fix
    end

    D -- OK --> E{"Region in<br/>role ARN correct?"}:::decision
    E -- Wrong --> E1["Fix the region<br/>segment of the ARN"]:::fix
    E -- OK --> F["Auth succeeds"]:::verify

Checklist & commands:

# workflow snippet — audience must match
- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/MyRole
    aws-region: us-east-1
    audience: sts.amazonaws.com   # must match trust policy

// IAM trust policy — subject must match branch/repo
{
  "Condition": {
    "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:org/repo:ref:refs/heads/main",
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
    }
  }
}

# verify OIDC provider thumbprint
aws iam list-open-id-connect-providers
aws iam get-open-id-connect-provider \
  --open-id-connect-provider-arn arn:aws:iam::ACCOUNT:oidc-provider/token.actions.githubusercontent.com

# test assume-role manually with a debug token
aws sts get-caller-identity

Prevention: Pin the aws-actions/configure-aws-credentials version (@v4 not @main). Store the role ARN in a repository variable not hardcoded in the workflow. Add a CI test workflow that fires on every trust-policy change PR and asserts aws sts get-caller-identity succeeds.

2. GitHub Actions: Job Hangs Indefinitely

Symptom: A step runs forever with no output; job never completes.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["Job hangs:<br/>step runs forever,<br/>no output"]:::err --> B{"Interactive<br/>prompt?"}:::decision
    B -- Yes --> B1["Add -y / --no-input<br/>flag to the command"]:::fix

    subgraph WAIT["Things silently blocking"]
        B -- No --> C{"sudo without<br/>NOPASSWD?"}:::decision
        C -- Yes --> C1["Add NOPASSWD in<br/>sudoers, or avoid sudo"]:::fix
        C -- No --> D{"Test waiting<br/>for a port?"}:::decision
        D -- Yes --> D1["Add a timeout or<br/>wait-on utility"]:::fix
        D -- No --> E{"Unreachable<br/>network host?"}:::decision
        E -- Yes --> E1["Mock or skip<br/>the network call"]:::fix
    end

    E -- No --> F{"job timeout-minutes<br/>not set?"}:::decision
    F -- Missing --> F1["Add timeout-minutes<br/>to the job"]:::fix
    F -- Set --> G["Cancel the job,<br/>check the last log line"]:::verify

Commands:

jobs:
  build:
    timeout-minutes: 15      # always set a ceiling
    steps:
      - run: apt-get install -y curl   # -y prevents prompt

# cancel a stuck run via CLI
gh run list --limit 5
gh run cancel <run-id>

# stream logs to find the last line before hang
gh run view <run-id> --log | tail -40

Prevention: Set timeout-minutes on every job and every step. Use continue-on-error: false (default) and add cancel-in-progress: true to concurrency groups so a new push cancels stuck old runs. Add a job_timeout metric via GitHub API to alert if any job exceeds 30 minutes.

3. ArgoCD: App Out of Sync But Won't Sync

Symptom: App shows OutOfSync in the UI but sync does nothing or errors immediately.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["App shows OutOfSync,<br/>sync does nothing<br/>or errors immediately"]:::err --> B["Check diff<br/>in ArgoCD UI"]:::verify
    B --> C{"Resource managed<br/>by another tool?"}:::decision
    C -- Yes --> C1["Remove Helm/kubectl<br/>annotations, or adopt<br/>the resource into ArgoCD"]:::fix

    subgraph SYNC["Sync configuration"]
        C -- No --> D{"Auto-sync<br/>disabled?"}:::decision
        D -- Yes --> D1["Enable auto-sync,<br/>or trigger manually"]:::fix
        D -- No --> E{"Sync error<br/>in k8s Events?"}:::decision
        E -- Yes --> E1["Fix the validation<br/>webhook error"]:::fix
    end

    E -- No --> F["Force sync<br/>with --force"]:::fix

Commands:

# inspect what ArgoCD thinks is different
argocd app diff my-app

# check sync status and last error
argocd app get my-app

# enable sync if disabled
argocd app set my-app --sync-policy automated

# force sync (overwrites live state)
argocd app sync my-app --force

# check for webhook rejections in k8s events
kubectl get events -n my-namespace --sort-by='.lastTimestamp' | tail -20

Prevention: Enable ArgoCD auto-sync with selfHeal: true and prune: true in non-prod environments so drift is corrected automatically. Use syncPolicy.syncOptions: [CreateNamespace=true] to avoid manual namespace creation. Add argocd app wait in the CD pipeline — fails the deploy pipeline if ArgoCD doesn't reach Synced+Healthy within the timeout.

4. ArgoCD: ImagePullBackOff After Deploy

Symptom: ArgoCD reports Synced/Healthy but pods stay in ImagePullBackOff.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["ArgoCD reports<br/>Synced/Healthy but pods<br/>stay ImagePullBackOff"]:::err --> B{"Tag exists<br/>in registry?"}:::decision
    B -- No --> B1["Push the correct tag,<br/>or fix the image ref"]:::fix
    B -- Yes --> C{"imagePullSecret<br/>in namespace?"}:::decision
    C -- Missing --> C1["Create the secret and<br/>add it to the serviceAccount"]:::fix

    subgraph AUTH["Registry auth"]
        C -- Present --> D{"ECR token<br/>expired?"}:::decision
        D -- Yes --> D1["Switch to IRSA,<br/>remove static creds"]:::fix
        D -- No --> E{"Image updater<br/>in use?"}:::decision
        E -- Yes --> E1["Check updater logs<br/>and annotation config"]:::fix
    end

    E -- No --> F["Check kubelet<br/>pull error details"]:::verify

Commands:

# check pod events for pull error details
kubectl describe pod <pod-name> -n <ns> | grep -A 10 Events

# verify image tag exists in ECR
aws ecr describe-images --repository-name my-repo \
  --image-ids imageTag=v1.2.3

# create ECR pull secret (short-term fix)
kubectl create secret docker-registry ecr-creds \
  --docker-server=<account>.dkr.ecr.<region>.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password)

# check argocd-image-updater logs
kubectl logs -n argocd deploy/argocd-image-updater | tail -50

Prevention: Use image digest pinning in ArgoCD (image: myapp@sha256:...) — eliminates tag races. Store imagePullSecrets as a sealed secret or External Secrets Operator resource, not a manually-created secret. Add a registry reachability check to the CD pipeline before deploying.

5. Jenkins Pipeline: Docker Build Fails in Agent

Symptom: docker: command not found or permission denied /var/run/docker.sock.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    A["Docker build fails:<br/>command not found /<br/>permission denied on socket"]:::err --> B{"docker binary<br/>present on agent?"}:::decision
    B -- No --> B1["Install Docker on<br/>the agent, or use DinD"]:::fix
    B -- Yes --> C{"jenkins user in<br/>docker group?"}:::decision
    C -- No --> C1["usermod -aG docker jenkins,<br/>restart the agent"]:::fix
    C -- Yes --> D{"Socket mounted<br/>in the agent pod?"}:::decision
    D -- No --> D1["Mount /var/run/docker.sock<br/>in the podTemplate"]:::fix

    subgraph APPROACH["Which build approach?"]
        D -- Yes --> E{"DinD sidecar<br/>or socket mount?"}:::decision
        E -- DinD --> E1["Use a privileged DinD<br/>sidecar container"]:::fix
        E -- "Socket mount" --> E2["Check socket perms:<br/>chmod 666 or group membership"]:::fix
    end

Pipeline snippet & commands:

// Jenkinsfile — socket mount approach
pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: docker
    image: docker:24-cli
    command: [sleep, infinity]
    volumeMounts:
    - name: docker-sock
      mountPath: /var/run/docker.sock
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock
"""
    }
  }
  stages {
    stage('Build') {
      steps {
        container('docker') {
          sh 'docker build -t my-image .'
        }
      }
    }
  }
}

// Jenkinsfile — Docker-in-Docker sidecar approach
pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: docker
    image: docker:24-cli
    command: [sleep, infinity]
    env:
    - name: DOCKER_HOST
      value: tcp://localhost:2375
  - name: dind
    image: docker:24-dind
    securityContext:
      privileged: true
    env:
    - name: DOCKER_TLS_CERTDIR
      value: ""
"""
    }
  }
  stages {
    stage('Build') {
      steps {
        container('docker') {
          sh 'docker build -t my-image .'
        }
      }
    }
  }
}

# on the agent node — add jenkins to docker group
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins

# verify socket permissions
ls -la /var/run/docker.sock   # should be srw-rw---- docker group

Prevention: Use rootless Docker or Kaniko/Buildah in CI agents — eliminates the docker.sock privilege escalation risk entirely. If Docker-in-Docker is required, use --privileged only in isolated ephemeral agents, never on long-lived shared agents.

6. Pipeline Deploys to Wrong Environment

Symptom: A staging branch push triggers a production deployment.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["Wrong env deployed:<br/>staging push triggers<br/>a production deploy"]:::err --> B{"Branch protection<br/>configured?"}:::decision
    B -- No --> B1["Add branch rules<br/>in GitHub/GitLab"]:::fix

    subgraph CONFIG["Environment wiring — check even after branch rules pass"]
        B -- Yes --> C{"Env var set<br/>correctly?"}:::decision
        C -- Wrong --> C1["Fix the ENV var in<br/>the workflow/Jenkinsfile"]:::fix
        C -- OK --> D{"Env name matches<br/>in workflow?"}:::decision
        D -- Mismatch --> D1["Align environment:<br/>name in the workflow"]:::fix
        D -- OK --> E{"ArgoCD app targets<br/>correct cluster/ns?"}:::decision
        E -- Wrong --> E1["Fix destination in<br/>the ArgoCD Application CR"]:::fix
        E -- OK --> F{"Correct Helm<br/>values file used?"}:::decision
        F -- Wrong --> F1["Fix the -f values<br/>path in sync config"]:::fix
    end

    F -- OK --> G["Add a manual<br/>approval gate for prod"]:::verify

Concrete fixes:

# GitHub Actions — gate production on branch + approval
jobs:
  deploy-prod:
    if: github.ref == 'refs/heads/main'
    environment: production          # requires manual approval in repo settings
    steps:
      - run: helm upgrade --install my-app ./chart -f values/prod.yaml

# verify ArgoCD app destination
argocd app get my-app -o json | jq '.spec.destination'

# check which values file ArgoCD is using
argocd app get my-app -o json | jq '.spec.source.helm'

Prevention: Use ArgoCD ApplicationSets with environment-specific values files and project RBAC — engineers can't promote to prod without approval. Add a diff step to the CD pipeline that shows what will change in each environment before applying. Use Helm --atomic flag so failed upgrades auto-rollback.

7. Container Image Build Passes but App Crashes in Staging

Symptom: docker build succeeds in CI, but container exits immediately in staging.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["docker build succeeds<br/>in CI, container exits<br/>immediately in staging"]:::err --> B{"Env vars<br/>missing?"}:::decision
    B -- Yes --> B1["Add vars to CI<br/>environment secrets"]:::fix
    B -- No --> C{"Secrets<br/>injected?"}:::decision
    C -- No --> C1["Mount secret, or use<br/>a secrets manager"]:::fix
    C -- Yes --> D{"Wrong base<br/>image arch?"}:::decision
    D -- "arm/amd mismatch" --> D1["Use a multi-arch build:<br/>docker buildx"]:::fix
    D -- OK --> E{"Health check<br/>path wrong?"}:::decision
    E -- Yes --> E1["Fix HEALTHCHECK or<br/>readinessProbe path"]:::fix
    E -- No --> F{"DB migration<br/>not run?"}:::decision
    F -- Yes --> F1["Add an init container,<br/>or a pre-deploy hook"]:::fix
    F -- No --> G["Check container<br/>logs in staging"]:::verify

Commands:

# pull and inspect the exact CI-built image locally
docker pull <registry>/my-app:<ci-tag>
docker run --rm -e ENV=staging <registry>/my-app:<ci-tag>

# check arch of built image
docker inspect <image> | jq '.[].Architecture'

# multi-arch build
docker buildx build --platform linux/amd64,linux/arm64 -t my-app:latest --push .

# check staging pod logs
kubectl logs -n staging deploy/my-app --previous
kubectl logs -n staging deploy/my-app -f

# describe pod for crash reason
kubectl describe pod -n staging -l app=my-app | grep -A 5 "Last State"

Prevention: Add a post-deploy smoke test step in the CD pipeline: kubectl rollout status deployment/my-app -n staging --timeout=120s && curl -f https://staging.example.com/health. If it fails, auto-rollback with kubectl rollout undo. Use progressDeadlineSeconds: 120 on Deployments so rollouts auto-fail fast.

8. Flaky Tests Blocking Pipeline

Symptom: Tests pass locally and sometimes in CI, but fail intermittently and block merges.

flowchart TD
    classDef err fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef decision fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#3498db,stroke:#2471a3,color:#fff

    A["Flaky test in CI:<br/>passes locally, fails<br/>intermittently, blocks merges"]:::err --> B

    subgraph CAUSES["Check causes in order"]
        B{"Race<br/>condition?"}:::decision
        B -- Yes --> B1["Run go test -race,<br/>fix the data race"]:::fix
        B -- No --> C{"Timing<br/>dependent?"}:::decision
        C -- Yes --> C1["Add retry or fix<br/>the deterministic wait"]:::fix
        C -- No --> D{"External service<br/>dependency?"}:::decision
        D -- Yes --> D1["Mock the service<br/>in tests"]:::fix
        D -- No --> E{"Parallel job<br/>resource contention?"}:::decision
        E -- Yes --> E1["Limit concurrency,<br/>or isolate resources"]:::fix
    end

    E -- No --> F["Increase test timeout:<br/>-timeout flag"]:::verify

Commands:

# detect races
go test -race ./...

# run with explicit timeout
go test -timeout 120s ./...

# re-run flaky test N times locally
for i in $(seq 1 10); do go test -run TestMyFlaky ./pkg/...; done

# run only failed tests from last run (requires gotestsum)
gotestsum --rerun-fails=3 --packages ./...

# GitHub Actions — retry flaky step
- name: Test
  uses: nick-fields/retry@v3
  with:
    timeout_minutes: 10
    max_attempts: 3
    command: go test -race -timeout 90s ./...

Prevention: Track flakiness rate per test in CI metrics — quarantine any test with >5% flakiness until fixed (don't just retry). Use go test -count=1 to disable test caching and always run fresh. For integration tests: use testcontainers-go to spin up real dependencies instead of mocks — eliminates a whole class of flakiness from mock state pollution.

Quick Reference