Kubernetes Production Scenarios

Twenty-plus Kubernetes failure patterns pulled from real on-call rotations — the symptom, the diagnostic commands, the cause tree, and the prevention that stops it recurring in the next incident.

CrashLoopBackOff: Debugging Runbook

CrashLoopBackOff means the container starts, crashes (exits non-zero), Kubernetes restarts it, it crashes again — and Kubernetes applies exponential backoff between restarts (10s → 20s → 40s → 80s → 160s → 5min cap). It will keep retrying indefinitely.

It is not a single cause — it's a symptom. The exit code tells you where to look first.

Step 1: Get the exit code and last logs

# See restart count and status
kubectl get pod <pod> -n <ns>

# Last crash reason + exit code
kubectl describe pod <pod> -n <ns>
# Look for: Last State, Exit Code, Reason (OOMKilled / Error / Completed)

# Logs from the crashed container (not the current one)
kubectl logs <pod> -n <ns> --previous

# If multi-container pod
kubectl logs <pod> -n <ns> -c <container-name> --previous

Step 2: Diagnose by exit code

Step 3: Work through the cause tree

flowchart TD
    classDef oom fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef apperr fill:#e67e22,stroke:#ba6018,color:#fff
    classDef binary fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef emptylog fill:#3498db,stroke:#2471a3,color:#fff
    classDef probe fill:#f39c12,stroke:#ba6018,color:#fff

    ROOT["CrashLoopBackOff"] --> OOM["Exit 137 — OOMKilled"]:::oom
    ROOT --> APP["Exit 1 — App panic / error"]:::apperr
    ROOT --> BIN["Exit 127 — binary not found"]:::binary
    ROOT --> EMPTY["Exit 1, but --previous<br/>logs are empty"]:::emptylog
    ROOT --> LIVE["Liveness probe killing an<br/>otherwise-healthy pod"]:::probe

    subgraph OOMBranch["OOM branch"]
        OOM --> OOM1["memory.limit too low<br/>→ increase resources.limits.memory"]:::oom
        OOM --> OOM2["memory leak<br/>→ profile with pprof, check goroutine count"]:::oom
        OOM --> OOM3["JVM heap not set<br/>→ add -Xmx flag"]:::oom
    end

    subgraph AppBranch["App-error branch"]
        APP --> APP1["Missing env var<br/>→ describe pod, check envFrom / env"]:::apperr
        APP --> APP2["Can't connect to DB/Redis on startup<br/>→ wrong SERVICE_NAME or port"]:::apperr
        APP --> APP3["Failed DB migration in same container<br/>→ move to a separate init container"]:::apperr
        APP --> APP4["Config file not found<br/>→ ConfigMap not mounted / wrong mountPath"]:::apperr
        APP --> APP5["Secret not found<br/>→ Secret doesn't exist in this namespace"]:::apperr
    end

    subgraph BinBranch["Binary/build branch"]
        BIN --> BIN1["Wrong command/args<br/>in Deployment spec"]:::binary
        BIN --> BIN2["Multi-stage build forgot<br/>to copy the binary"]:::binary
    end

    subgraph EmptyBranch["Empty-log branch"]
        EMPTY --> EMPTY1["App crashes before logger initializes<br/>→ add stderr logging early"]:::emptylog
        EMPTY --> EMPTY2["Init container failing<br/>→ kubectl logs -c init-container"]:::emptylog
        EMPTY --> EMPTY3["readinessProbe killing pod<br/>before app fully starts"]:::emptylog
    end

    subgraph LiveBranch["Liveness-probe branch"]
        LIVE --> LIVE1["initialDelaySeconds too short<br/>→ app not ready when probe fires"]:::probe
        LIVE --> LIVE2["Probe endpoint wrong<br/>→ 404 returns, pod killed"]:::probe
        LIVE --> LIVE3["timeoutSeconds too low<br/>→ slow startup looks like failure"]:::probe
    end

Common contributors and fixes

1. Missing / wrong environment variable

kubectl describe pod <pod> -n <ns> | grep -A5 "Environment"
# Or check what the app is actually seeing:
kubectl exec <pod> -n <ns> -- env | grep DB_

# Fix: check ConfigMap and Secret refs in Deployment

# Common mistake: referencing a Secret that doesn't exist in this namespace
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret      # does this Secret exist in the same namespace?
      key: password

kubectl get secret db-secret -n <ns>   # 404 here = pod will CrashLoop

2. Can't connect to dependency on startup

App tries to connect to DB/Redis/external API during init() or startup, fails, panics.

# Test connectivity from inside a debug pod in same namespace
kubectl run debug --rm -it --image=busybox -n <ns> -- sh
wget -qO- http://my-service:5432   # or nc -zv my-service 5432

Fix: Add startup retry logic — don't fail fast on first connection attempt. Or use an init container to wait for the dependency:

initContainers:
- name: wait-for-db
  image: busybox
  command: ['sh', '-c', 'until nc -z postgres-svc 5432; do echo waiting; sleep 2; done']

3. OOMKilled — memory limit too low

kubectl describe pod <pod> | grep -A3 "Last State"
# Last State: Terminated  Reason: OOMKilled

# Check current memory usage before the crash
kubectl top pod <pod> -n <ns>

# Check limits
kubectl get pod <pod> -o jsonpath='{.spec.containers[0].resources}'

Fix:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"   # increase this, or remove limit to use node memory

For Go apps: GOGC env var controls GC aggressiveness. GOMEMLIMIT (Go 1.19+) sets a soft memory ceiling before GC kicks in aggressively:

env:
- name: GOMEMLIMIT
  value: "450MiB"   # slightly below the k8s limit

4. Liveness probe killing the pod

Pod shows CrashLoopBackOff but logs look fine — the liveness probe is the killer.

kubectl describe pod <pod> | grep -A10 "Liveness"
# Events: Liveness probe failed: ... Killing container with id...

# Fix: give app time to start before first probe
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30    # wait 30s before first check
  periodSeconds: 10
  failureThreshold: 3        # need 3 consecutive failures before kill
  timeoutSeconds: 5          # give it 5s to respond

5. Init container failing silently

# Init containers show separately
kubectl get pod <pod> -n <ns>
# Init:0/1 means init container hasn't completed

kubectl logs <pod> -n <ns> -c <init-container-name>
kubectl logs <pod> -n <ns> -c <init-container-name> --previous

6. Wrong image / wrong arch

kubectl describe pod <pod> | grep -A5 "Events"
# exec format error → image built for wrong arch (amd64 image on arm node or vice versa)

# Fix: build multi-arch image
docker buildx build --platform linux/amd64,linux/arm64 -t my-org/app:v1 --push .

Quick checklist

□ kubectl logs <pod> --previous          → read the actual crash message
□ kubectl describe pod <pod>             → exit code, events, probe config
□ kubectl get events -n <ns> --sort-by=.lastTimestamp  → cluster-level events
□ exit code 137?                         → OOMKilled, check memory limits
□ exit code 1, empty logs?               → init container? probe killing it?
□ env vars correct?                      → describe pod, check secret/configmap exists
□ can pod reach its dependencies?        → debug pod with nc/wget
□ liveness probe initialDelaySeconds?    → might be firing too early
□ init containers healthy?               → kubectl logs -c <init-container>
□ image right arch?                      → exec format error in events

Prevention: Set resources.limits.memory on all containers — OOMKills become predictable, not random. Use GOMEMLIMIT for Go apps. Add startup probes with failureThreshold: 30 so slow-starting apps don't get killed by liveness. In CI: run kubectl apply --dry-run=server to catch missing secrets/configmaps before deploy. Use init containers for dependency checks instead of fast-failing in the main container.

Pod Stuck in Pending

Pod is created but never scheduled — it's sitting in the scheduler queue with no node assigned.

kubectl describe pod <pod> -n <ns>
# Look at Events section at the bottom — FailedScheduling with reason

Cause tree

# Check actual node capacity vs allocatable
kubectl describe nodes | grep -A5 "Allocated resources"

# Check what's consuming resources
kubectl top nodes
kubectl top pods -A --sort-by=memory

# Check if cluster autoscaler is blocked
kubectl logs -n kube-system -l app=cluster-autoscaler | tail -50

Prevention: Set accurate resources.requests — oversized requests cause artificial pending. Use Cluster Autoscaler with proper minSize/maxSize and ensure node groups cover all required taints/labels. Add podAntiAffinity with preferredDuringScheduling (not required) to avoid unsatisfiable constraints. Test scheduling with kubectl apply --dry-run=server.

All Replicas Scheduled on One Node — Node Dies — Full Outage

Symptom: A Deployment with replicas: 3 (or any N > 1) has all replicas running on the same node. That node dies (hardware fault, kernel panic, host-level EC2 issue). All N pods disappear simultaneously — a full outage, not a partial degradation.

kubectl get pods -l app=<name> -o wide
# NAME            NODE
# my-app-abc123   node-a   ← all 3 rows show node-a — this is the exposure, even before node-a dies
# my-app-def456   node-a
# my-app-ghi789   node-a

Why this happens even though it "shouldn't": the default Kubernetes scheduler has no anti-affinity behavior by default. Its Score plugins (LeastAllocated/ImageLocality — see scheduler-internals.md) optimize for bin-packing and resource fit, not spread. Nothing in the default algorithm prevents multiple replicas of the same Deployment from landing on the same node — if that node scores highest each time (common right after a batch of pods is created together, or on a lightly-loaded cluster), the scheduler will legitimately place all of them there. This is not a bug; it's the absence of a rule you have to add explicitly.

flowchart TD
    classDef config fill:#3498db,stroke:#2471a3,color:#fff
    classDef place fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef fail fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef good fill:#27ae60,stroke:#1e8449,color:#fff

    A["Deployment: replicas: 3,<br/>no anti-affinity / spread configured"]:::config --> B

    subgraph EXPOSURE["Scheduler concentrates all 3 pods —<br/>nothing in the default algorithm prevents this"]
        B["Scheduler places pod 1<br/>→ Node A (best bin-pack score)"]:::place --> C["Scheduler places pod 2<br/>→ Node A again<br/>(still best score, no rule against it)"]:::place
        C --> D["Scheduler places pod 3<br/>→ Node A again"]:::place
    end

    D --> E["Node A dies<br/>(hardware fault / kernel panic / host issue)"]:::fail
    E --> F["All 3/3 pods gone simultaneously<br/>= FULL OUTAGE, not degradation"]:::fail
    F -.->|"contrast"| G["With topologySpreadConstraints:<br/>losing Node A only drops to 2/3 —<br/>degraded, not down"]:::good

Cause tree

The fix — make single-node concentration structurally impossible

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      topologySpreadConstraints:
      # Spread across NODES — the direct fix for "3/3 on one node"
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule    # hard — refuse to co-locate beyond skew 1
        labelSelector:
          matchLabels: {app: my-app}
      # Spread across AZs too — protects against a whole-AZ failure, not just one node
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels: {app: my-app}

maxSkew: 1 with topologyKey: kubernetes.io/hostname caps the difference in pod count between the most- and least-loaded node at 1. With 3 replicas and 3+ available nodes, this forces one pod per node — losing any single node now drops you to 2/3, not 0/3.

Older/equivalent mechanism — pod anti-affinity (still common in existing manifests; topologySpreadConstraints is the modern preferred API):

spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels: {app: my-app}
            topologyKey: kubernetes.io/hostname   # hard: never co-locate two replicas on the same node

Why DoNotSchedule (hard), not ScheduleAnyway (soft), for mission-critical services: with the soft form, the scheduler prefers spreading but will still co-locate replicas if it has to — silently reintroducing the exact single-point-of-failure this is meant to prevent, with no warning at all. The hard form instead leaves a pod Pending if it truly can't satisfy the spread — which surfaces as a visible, alertable scheduling problem you can fix by adding capacity, instead of a silent landmine that only detonates when the node actually dies.

This must be paired with cluster capacity, or it just creates Pending pods instead:

□ At least as many schedulable nodes as replicas (3 replicas needs >= 3 nodes with free capacity)
□ Cluster Autoscaler / Karpenter can add nodes if the hard constraint currently can't be satisfied
□ PodDisruptionBudget (minAvailable) so a voluntary drain doesn't undo the spread by
  evicting 2 of 3 already-spread pods back onto the same remaining node
□ replicas >= 3 for mission-critical services — with 2 replicas, losing 1 node is
  already a 50% capacity loss even with perfect spread

Detecting this before (or after) it bites you

# Where are a Deployment's replicas actually scheduled right now?
kubectl get pods -l app=my-app -o wide

# Fleet-wide audit: flag any app with >1 replica but only 1 distinct node in use
kubectl get pods -A -o json | jq -r '
  .items | group_by(.metadata.labels.app) |
  map({app: .[0].metadata.labels.app, nodes: (map(.spec.nodeName) | unique)}) |
  map(select((.nodes | length) == 1)) '

Prevention: Treat topologySpreadConstraints as a default, not opt-in, for every Deployment with more than 1 replica that matters for availability — enforce it fleet-wide via a Kyverno/OPA mutating policy so teams can't accidentally ship without it. Re-run the jq audit periodically (or wire it into a policy-as-code check), since a Deployment can drift into single-node concentration again after a later reschedule/drain even if it was correctly spread at initial rollout.

ImagePullBackOff / ErrImagePull

Container image can't be pulled. ErrImagePull is the first attempt; ImagePullBackOff is repeated failure with backoff.

kubectl describe pod <pod> -n <ns>
# Events: Failed to pull image "...", reason in the message

Cause tree

# Test pulling manually on the node
kubectl debug node/<node-name> -it --image=busybox
# inside: docker pull <image> or crictl pull <image>

# Check if imagePullSecret exists in the right namespace
kubectl get secret regcred -n <ns>

# For ECR: check node IAM role has ecr:GetAuthorizationToken
aws iam simulate-principal-policy \
  --policy-source-arn <node-role-arn> \
  --action-names ecr:GetAuthorizationToken

Prevention: Use image digest pinning (image: my-app@sha256:abc123) instead of mutable tags in production — guarantees the same image always. Set up ECR pull-through cache or mirror to avoid rate limits. For ECR: use IRSA on the node role with ecr:GetAuthorizationToken + ecr:BatchGetImage. Test image pulls in CI before deploying.

Pod Stuck in Terminating

kubectl delete pod was run but pod stays in Terminating state indefinitely.

kubectl describe pod <pod> -n <ns>
# Look for: finalizers, volumes not unmounting, preStop hook hanging

Causes

1. Finalizer not being removed

kubectl get pod <pod> -n <ns> -o json | jq '.metadata.finalizers'
# If a controller crashed and never removed the finalizer, pod hangs

# Force remove finalizer (only if you're sure the controller is gone)
kubectl patch pod <pod> -n <ns> -p '{"metadata":{"finalizers":[]}}' --type=merge

2. preStop hook hanging

# preStop has a hard deadline of terminationGracePeriodSeconds (default 30s)
# If hook runs longer, pod is force killed after the grace period
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]  # must complete within grace period

3. Volume not unmounting (PVC)

kubectl describe pod <pod> | grep -A5 "Volumes"
# "Unable to unmount volumes" in events = storage driver issue

# Force delete as last resort (may leave volume in bad state)
kubectl delete pod <pod> -n <ns> --grace-period=0 --force

4. Node is NotReady / unreachable

# If node went offline, pods on it stay Terminating until node comes back or is deleted
kubectl get node <node>
kubectl delete node <node>   # removes the node object, pods get rescheduled

Prevention: Avoid finalizers unless necessary — they're the #1 cause of stuck Terminating pods. Set terminationGracePeriodSeconds to a realistic value (preStop duration + drain time + buffer). For node failures: enable NonGracefulNodeShutdown feature gate (GA in 1.28) so stuck Terminating pods are auto-cleaned up.

Node NotReady

kubectl get nodes
# NAME        STATUS      ROLES    AGE
# node-1      NotReady    <none>   2d

kubectl describe node <node-name>
# Look at: Conditions, Events, kubelet logs

Cause tree

flowchart TD
    classDef proc fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef disk fill:#e67e22,stroke:#ba6018,color:#fff
    classDef mem fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef pid fill:#f39c12,stroke:#ba6018,color:#fff
    classDef net fill:#3498db,stroke:#2471a3,color:#fff
    classDef cloud fill:#7f8c8d,stroke:#616a6b,color:#fff

    ROOT["Node NotReady"] --> KUBELET["kubelet stopped"]:::proc
    ROOT --> DISK["Disk pressure<br/>(DiskPressure=True)"]:::disk
    ROOT --> MEM["Memory pressure<br/>(MemoryPressure=True)"]:::mem
    ROOT --> PID["PID pressure<br/>(PIDPressure=True)"]:::pid
    ROOT --> NET["Network unreachable"]:::net
    ROOT --> CLOUD["Cloud provider issue (EKS)"]:::cloud

    KUBELET --> KUBELET1["ssh to node:<br/>systemctl status kubelet,<br/>journalctl -u kubelet -n 100"]:::proc

    subgraph DiskBranch["Disk pressure"]
        DISK --> DISK1["Node full of logs/images<br/>→ kubelet evicts pods"]:::disk
        DISK1 --> DISK2["Fix: kubectl drain + increase disk,<br/>or add an image-pruner CronJob<br/>(crictl rmi --prune)"]:::disk
    end

    subgraph MemBranch["Memory pressure"]
        MEM --> MEM1["System processes<br/>consuming memory"]:::mem
        MEM1 --> MEM2["Fix: kubectl drain + investigate,<br/>check for a leak in DaemonSets"]:::mem
    end

    subgraph PidBranch["PID pressure"]
        PID --> PID1["Too many processes<br/>(fork bomb, runaway threads)"]:::pid
        PID1 --> PID2["Fix: find the pod —<br/>kubectl top pods --sort-by=cpu"]:::pid
    end

    subgraph NetBranch["Network unreachable"]
        NET --> NET1["CNI plugin crashed<br/>→ pods can't get IPs"]:::net
        NET1 --> NET2["kubectl logs -n kube-system<br/>-l k8s-app=aws-node (VPC CNI)<br/>or -l app=calico-node"]:::net
    end

    CLOUD --> CLOUD1["EC2 instance health check failing<br/>→ terminate + replace node<br/>aws ec2 describe-instance-status"]:::cloud

# Cordon + drain to safely remove workloads before investigating
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# Check node events
kubectl get events -n default --field-selector involvedObject.name=<node>

Prevention: Monitor node conditions with kube_node_status_condition Prometheus metric. Alert on DiskPressure=True before pods are evicted. Add image GC thresholds in kubelet config (imageGCHighThresholdPercent: 80). Use node health checks with AWS Auto Scaling Group health check integration so unhealthy nodes are automatically replaced.

High Latency / Slow Requests

Pod is Running, no crashes, but requests are slow.

Step 1 — isolate the layer

# Is it all pods or specific ones?
kubectl top pods -n <ns>    # CPU throttled? Memory pressure?

# Check if requests hit a specific pod (check per-pod metrics in Grafana/Datadog)
# Is load balancer distributing unevenly?

Step 2 — CPU throttling (most common invisible cause)

# Check throttling metrics
kubectl top pod <pod> -n <ns>
# If CPU is at limit constantly → throttled

# In Prometheus:
# container_cpu_cfs_throttled_seconds_total — cumulative throttle time
# rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])
# > 0.25 means 25%+ of scheduling periods throttled → very bad for latency

Fix: Either raise the CPU limit, or remove it entirely (keep only requests). CPU limits cause p99 latency spikes even when average CPU is low.

resources:
  requests:
    cpu: "500m"
  limits:
    cpu: "2000m"   # raise this, or remove limits entirely for latency-sensitive services

Step 3 — GC pauses (Go / JVM)

# Go: check GC stats via pprof
kubectl port-forward pod/<pod> 6060:6060
curl http://localhost:6060/debug/pprof/heap > heap.prof
go tool pprof heap.prof

# Check GC frequency
curl http://localhost:6060/debug/vars | jq '.memstats'
# NumGC high + PauseTotalNs high → GC pressure

Fix for Go: Set GOMEMLIMIT to give GC headroom before hitting k8s limit.

Step 4 — connection pool exhaustion

# Symptoms: latency spikes at high concurrency, logs show "connection wait timeout"
# App is waiting for a DB connection from the pool

# Check pool metrics if exposed, or:
kubectl exec <pod> -- cat /proc/<pid>/net/tcp | wc -l   # open TCP connections

Fix: increase pool size, or add read replicas / RDS Proxy.

Step 5 — DNS resolution slow

# DNS latency inside cluster
kubectl run dnstest --rm -it --image=busybox -- sh
time nslookup my-service.my-namespace.svc.cluster.local

# CoreDNS performance
kubectl top pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns | grep -i error

Fix: use fully qualified domain names (FQDN) to avoid search domain iteration, or tune CoreDNS replicas.

Prevention: Never set CPU limits on latency-sensitive services — use requests only. Alert on container_cpu_cfs_throttled_periods_total / container_cpu_cfs_periods_total > 0.25. Set dnsConfig.options: [{name: ndots, value: "2"}] on all pods. Use GOMEMLIMIT for Go services. Deploy NodeLocal DNSCache to eliminate DNS as a latency source.

PVC Stuck in Pending

kubectl get pvc -n <ns>
# NAME    STATUS    VOLUME   CAPACITY   STORAGECLASS
# data    Pending                       gp3

kubectl describe pvc data -n <ns>
# Events tell you why

# Check CSI driver (EBS on EKS)
kubectl logs -n kube-system -l app=ebs-csi-controller -c ebs-plugin | tail -50

# Check storage classes
kubectl get storageclass

# Check if IAM role for CSI has ebs:CreateVolume permission

Prevention: Use WaitForFirstConsumer volumeBindingMode on StorageClasses — prevents PVCs binding to the wrong AZ before the pod is scheduled. Set ResourceQuota for storage to prevent runaway PVC creation. Test CSI driver RBAC with kubectl auth can-i in staging before deploying new clusters.

Resource Quota / LimitRange Blocking Pods

# Pod fails to create with "exceeded quota" or "must specify limits"
kubectl describe quota -n <ns>
kubectl describe limitrange -n <ns>

# Example output:
# Resource         Used    Hard
# requests.cpu     1800m   2000m    ← close to limit
# limits.memory    3Gi     4Gi

LimitRange sets defaults and min/max for containers in a namespace. If a pod has no resources set and LimitRange requires it, the pod is rejected.

# See what defaults are being injected
kubectl get limitrange -n <ns> -o yaml

Prevention: Set LimitRange defaults in every namespace so pods without resource specs still get sane defaults. Alert on kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.80 to catch quota exhaustion before it blocks deployments. Document quota allocations per team in runbooks.

Quick Reference: All Pod States

Pod Running But Requests Failing Silently

Symptom: Pod is Running and Ready, but requests return errors or wrong data. Readiness probe passes but the app is broken inside.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["Pod shows Running + Ready,<br/>probe is passing"]:::start --> B["curl the app directly<br/>from inside the pod"]:::check
    B --> C{"Response actually<br/>correct?"}:::check
    C -- No --> D["Check app logs"]:::check
    D --> E{"Error visible<br/>in logs?"}:::check
    E -- Yes --> F["Fix app / config"]:::fix
    E -- No --> G["Probe path ≠ real health path —<br/>the probe is lying about health"]:::trap
    G --> H["Check env vars / ConfigMap"]:::check
    H --> I["Check upstream dependency"]:::check
    C -- Yes --> J["Issue is external —<br/>ingress / Service layer, not the pod"]:::fix

Commands:

# exec into pod and curl the app directly
kubectl exec -it <pod> -- curl -v http://localhost:<port>/healthz

# check app logs
kubectl logs <pod> --tail=100

# check readiness probe config
kubectl describe pod <pod> | grep -A 10 "Readiness"

# check env vars
kubectl exec -it <pod> -- env | grep -i db
kubectl exec -it <pod> -- env | grep -i api

Expected output snippet:

Readiness:  http-get http://:8080/ready delay=5s timeout=1s period=10s
# If /ready returns 200 but app is broken, the probe path is wrong

Root Cause Table:

Prevention: Separate liveness from readiness probes — readiness should check actual app health (e.g. DB ping), liveness should only check if the process is hung. Use /readyz for readiness (checks dependencies) and /livez for liveness (just checks process). Add integration tests in CI that deploy to staging and verify the probe endpoints return correct status codes.

DNS Resolution Failures Inside Pods

Symptom: Pod gets NXDOMAIN or connection timeout when resolving service names. nslookup my-svc fails inside the pod.

sequenceDiagram
    participant P as Pod
    participant CD as CoreDNS
    participant S as Service object
    participant E as Endpoints/EndpointSlice

    Note over P,CD: Check 1 — is CoreDNS even reachable?
    P->>CD: nslookup my-svc (via resolv.conf search domains)
    rect rgb(80, 40, 40)
    CD-->>P: NXDOMAIN, or timeout entirely
    Note over P: If this fails, check CoreDNS pod status<br/>and NetworkPolicy egress to port 53 first
    end

    Note over CD,S: Check 2 — does the Service exist?
    CD->>S: Resolve my-svc.default.svc.cluster.local
    rect rgb(80, 40, 40)
    S-->>CD: Service not found in that namespace
    Note over S: kubectl get svc — create it, or fix the namespace
    end

    Note over S,E: Check 3 — does the Service have live backends?
    S->>E: Look up ready backend pod IPs
    rect rgb(80, 40, 40)
    E-->>S: Endpoints list is empty
    Note over E: Compare svc selector vs pod labels,<br/>and confirm pods are actually Ready
    end

    rect rgb(30, 70, 45)
    CD-->>P: Healthy path — returns ClusterIP
    end

Commands:

# test DNS from inside pod
kubectl exec -it <pod> -- nslookup my-svc
kubectl exec -it <pod> -- nslookup my-svc.default.svc.cluster.local

# check resolv.conf
kubectl exec -it <pod> -- cat /etc/resolv.conf
# expect: search default.svc.cluster.local svc.cluster.local cluster.local

# is CoreDNS running?
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Root Cause Table:

Prevention: Deploy NodeLocal DNSCache DaemonSet — each node caches DNS locally, eliminating CoreDNS as a SPOF. Set ndots: 2 in pod dnsConfig to reduce DNS round trips. Alert on coredns_dns_responses_total{rcode="SERVFAIL"} and CoreDNS OOMKilled. Scale CoreDNS replicas proportional to cluster size (1 replica per 16 nodes minimum).

NetworkPolicy Blocking Traffic

Symptom: Pod A can't reach Pod B or a Service, but both are Running. No obvious error — just connection refused or timeout.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["Pod A → Pod B fails<br/>(no obvious error in either pod)"]:::start --> B["kubectl exec pod-a --<br/>curl pod-b"]:::check
    B --> C{"Timeout,<br/>or connection refused?"}:::check

    subgraph TimeoutBranch["Timeout — usually a policy problem"]
        C -- Timeout --> D["Check NetworkPolicy<br/>objects exist at all"]:::check
        D --> E["Check ingress rules<br/>on destination pod"]:::check
        E --> F{"Labels match<br/>podSelector?"}:::check
        F -- No --> G["Fix policy selector<br/>or pod labels"]:::fix
        F -- Yes --> H["Check egress rules<br/>on source pod"]:::check
        H --> I{"Egress allows<br/>the destination port?"}:::check
        I -- No --> J["Add the missing<br/>egress rule"]:::fix
        I -- Yes --> K["CNI plugin doesn't enforce<br/>NetworkPolicy at all"]:::trap
    end

    C -- Refused --> L["Not a policy issue —<br/>app isn't listening on that port"]:::trap

Commands:

# list all network policies in namespace
kubectl get networkpolicy -n <ns>

# inspect a policy
kubectl describe networkpolicy <policy-name> -n <ns>

# test connectivity from pod A
kubectl exec -it <pod-a> -- curl -v http://<pod-b-ip>:<port>

# use ephemeral debug container (k8s 1.23+)
kubectl debug -it <pod-a> --image=nicolaka/netshoot --target=<container>
# then: curl, nslookup, tcpdump

Root Cause Table:

Prevention: Adopt a default-deny-all NetworkPolicy per namespace and explicitly allowlist required traffic. Test policies with kubectl exec -- curl in staging before prod. Use Cilium's hubble observe --verdict DROPPED to see what's being blocked in real time. Include NetworkPolicy in service deployment manifests so policies travel with the service.

Service Not Routing to Pods (Empty Endpoints)

Symptom: Service exists and has a ClusterIP, but requests get connection refused or no backends respond.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["Service has a ClusterIP,<br/>but requests fail"]:::start --> B["kubectl get endpoints my-svc"]:::check
    B --> C{"ENDPOINTS<br/>column empty?"}:::check

    subgraph EmptyBranch["Endpoints empty — selector/readiness problem"]
        C -- Yes --> D["Compare svc selector<br/>vs pod labels"]:::check
        D --> E{"Labels<br/>match?"}:::check
        E -- No --> F["Fix labels or selector"]:::fix
        E -- Yes --> G["Check pod Ready condition"]:::check
        G --> H{"Pod Ready=True?"}:::check
        H -- No --> I["Fix readiness probe<br/>or app startup"]:::fix
        H -- Yes --> J["Check targetPort vs<br/>containerPort"]:::check
        J --> K{"Ports match?"}:::check
        K -- No --> L["Fix targetPort in svc"]:::fix
        K -- Yes --> M["Check pod namespace<br/>matches svc namespace"]:::trap
    end

    C -- No --> N["Endpoints exist —<br/>check kube-proxy/iptables rules<br/>on the node instead"]:::trap

Commands:

# check endpoints
kubectl get endpoints my-svc
# ENDPOINTS column should not be "<none>"

# describe the service to see selector
kubectl describe svc my-svc

# check pod labels
kubectl get pods --show-labels

# verify targetPort
kubectl get svc my-svc -o jsonpath='{.spec.ports[*].targetPort}'

Expected output snippet:

NAME      ENDPOINTS         AGE
my-svc    <none>            5m   ← problem: no backends

Root Cause Table:

Prevention: Use label conventions enforced by LimitRange/OPA — all pods must have app label. Write CI checks (conftest, kyverno) that validate Service selectors match Deployment labels before merge. Add a Prometheus alert on kube_endpoint_address_not_ready > 0 sustained for > 2 minutes.

HPA Not Scaling

Symptom: Load is high but HPA stays at minReplicas. kubectl get hpa shows <unknown> for current metrics.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["kubectl get hpa shows<br/>&lt;unknown&gt; for TARGETS"]:::start --> B["Is metrics-server<br/>even running?"]:::check
    B --> C{"kubectl top pods<br/>works?"}:::check
    C -- No --> D["Install / fix<br/>metrics-server"]:::fix

    subgraph ReqBranch["metrics-server is fine — check the pod spec"]
        C -- Yes --> E["Does the pod have<br/>resources.requests set?"]:::check
        E --> F{"requests.cpu<br/>present?"}:::check
        F -- No --> G["Add resource requests —<br/>HPA can't compute % of nothing"]:::trap
    end

    subgraph CustomBranch["Custom/external metrics path (KEDA, prometheus-adapter)"]
        F -- Yes --> H["Scaling on a custom<br/>or external metric?"]:::check
        H --> I["prometheus-adapter /<br/>KEDA pod running?"]:::check
        I --> J["Check HPA's<br/>externalMetrics config"]:::check
        J --> K["Check RBAC for the<br/>custom metrics API"]:::check
        K --> L["kubectl get apiservices |<br/>grep metrics.k8s.io"]:::fix
    end

Commands:

# check HPA status
kubectl get hpa
kubectl describe hpa <hpa-name>

# verify metrics-server is reachable
kubectl top pods -n <ns>
kubectl top nodes

# check metrics API registration
kubectl get apiservices | grep metrics

# check if resource requests are set on pods
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].resources}'

Expected output snippet:

NAME   REFERENCE         TARGETS         MINPODS  MAXPODS  REPLICAS
web    Deployment/web    <unknown>/50%   2        10       2
# <unknown> means metrics pipeline is broken

Root Cause Table:

Prevention: Always set resources.requests.cpu — HPA silently shows <unknown> without it. Install metrics-server as part of cluster bootstrap (not optional). For production: use KEDA with external metrics (queue depth, RPS) instead of CPU-only HPA — CPU is a lagging indicator of load.

Deployment Stuck in Rollout

Symptom: kubectl rollout status hangs. New pods won't start (CrashLoop/Pending) or old pods won't terminate.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef newp fill:#e67e22,stroke:#ba6018,color:#fff
    classDef oldp fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["kubectl rollout status hangs —<br/>new ReplicaSet never reaches desired count"]:::start --> B["Check new pods"]:::newp
    A --> F["Check old pods"]:::oldp

    subgraph NewBranch["New ReplicaSet not coming up"]
        B --> C{"Pending or<br/>CrashLoopBackOff?"}:::newp
        C -- Pending --> D["Check new pod events —<br/>scheduling issue (see Pending scenario)"]:::newp
        C -- CrashLoop --> E["kubectl logs new-pod —<br/>bad image/config (see CrashLoop runbook)"]:::newp
    end

    subgraph OldBranch["Old ReplicaSet not scaling down"]
        F --> G{"Old pods stuck<br/>Terminating?"}:::oldp
        G -- Yes --> H["Check preStop hook /<br/>finalizers"]:::oldp
        H --> I["kubectl patch to<br/>remove finalizer"]:::fix
        G -- No --> J["Check maxUnavailable /<br/>maxSurge settings"]:::oldp
        J --> K["Check PodDisruptionBudget<br/>blocking the drain"]:::trap
        K --> L["kubectl get pdb —<br/>minAvailable too strict?"]:::fix
    end

Commands:

# check rollout status
kubectl rollout status deployment/<name>

# rollout history
kubectl rollout history deployment/<name>

# inspect new and old pods
kubectl get pods -l app=<name> --sort-by=.metadata.creationTimestamp

# describe a stuck pod
kubectl describe pod <new-pod>

# check PodDisruptionBudget
kubectl get pdb -n <ns>
kubectl describe pdb <pdb-name>

# rollback if needed
kubectl rollout undo deployment/<name>

Root Cause Table:

Prevention: Set a progressDeadlineSeconds (default 600s) — rollout auto-fails if new pods don't become Ready within that window, making CI pipelines fail fast. Always set a PDB with minAvailable: 1 so rollouts can't kill all replicas. Add a readiness probe that fails until the app is truly ready (not just started).

Webhook Admission Failures

Symptom: kubectl apply returns an admission webhook error. Resource can't be created or updated.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef trap fill:#e74c3c,stroke:#c0392b,color:#fff

    A["kubectl apply fails —<br/>admission webhook error"]:::start --> B["API server calls<br/>MutatingWebhook chain"]:::check
    B --> C["API server calls<br/>ValidatingWebhook chain"]:::check
    C --> D{"Webhook pod<br/>Running?"}:::check
    D -- No --> E["Fix webhook<br/>deployment"]:::fix
    D -- Yes --> F{"Cert valid?<br/>caBundle correct?"}:::check
    F -- No --> G["Renew cert /<br/>update caBundle"]:::fix

    subgraph PolicyBranch["failurePolicy decides the blast radius"]
        F -- Yes --> H{"failurePolicy<br/>Fail or Ignore?"}:::check
        H -- Fail --> I["Webhook timeout or crash<br/>BLOCKS every matching apply<br/>cluster-wide, including the fix"]:::trap
        H -- Ignore --> J["Webhook silently skipped —<br/>request admitted unchecked"]:::check
        I --> K["Check webhook<br/>Service + port + endpoints"]:::fix
    end

Commands:

# list webhook configs
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# inspect a webhook
kubectl describe validatingwebhookconfiguration <name>

# check webhook pod and service
kubectl get pods -n <webhook-ns>
kubectl get svc -n <webhook-ns>

# test webhook service reachability
kubectl exec -it <debug-pod> -- curl -k https://<webhook-svc>.<ns>.svc/validate

Root Cause Table:

Prevention: Set failurePolicy: Ignore on all non-critical webhooks (security webhooks can be Fail). Add namespaceSelector to exclude kube-system from webhook scope — prevents the webhook from blocking its own recovery. Use cert-manager to auto-rotate webhook TLS certs. Run webhook pods with PDB and 2+ replicas.

Node Disk Pressure Evicting Pods

Symptom: Pods suddenly evicted with reason: Evicted. Node shows DiskPressure=True condition.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef nodefs fill:#e67e22,stroke:#ba6018,color:#fff
    classDef imagefs fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    A["Pods evicted with reason: Evicted —<br/>node shows DiskPressure=True"]:::start --> B["kubectl describe node —<br/>which filesystem tripped it?"]:::start
    B --> C{"nodefs or<br/>imagefs threshold?"}:::start

    subgraph NodefsBranch["nodefs — kubelet's own working directory"]
        C -- nodefs --> D["df -h /var/lib/kubelet<br/>on the node"]:::nodefs
        D --> F{"Logs filling<br/>the disk?"}:::nodefs
        F -- Yes --> G["Fix log rotation /<br/>reduce log verbosity"]:::fix
        F -- No --> H{"emptyDir volumes<br/>growing unbounded?"}:::nodefs
        H -- Yes --> H2["Add sizeLimit to<br/>emptyDir specs"]:::fix
    end

    subgraph ImagefsBranch["imagefs — container image/layer storage"]
        C -- imagefs --> E["du -sh /var/lib/containerd<br/>on the node"]:::imagefs
        E --> I["crictl rmi --prune —<br/>remove unused image layers"]:::fix
    end

    H -- No --> J["Neither — disk is just<br/>too small: expand or add<br/>a larger node group"]:::fix

Commands:

# find evicted pods
kubectl get pods --field-selector=status.phase=Failed -A | grep Evicted

# describe node for disk pressure
kubectl describe node <node-name> | grep -A 5 "Conditions"

# debug node filesystem (k8s 1.23+)
kubectl debug node/<node-name> -it --image=ubuntu
# inside: df -h, du -sh /var/log/containers/*

# clean up evicted pods
kubectl delete pods --field-selector=status.phase=Failed -A

Expected output snippet:

Conditions:
  Type            Status  ...
  DiskPressure    True    kubelet has disk pressure

Root Cause Table:

Prevention: Set kubelet --container-log-max-size=50Mi --container-log-max-files=3. Configure image GC thresholds (imageGCHighThresholdPercent: 80). Alert on node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.15. Add sizeLimit to all emptyDir volumes. Use instance types with ≥ 100Gi root volumes for nodes.

etcd Slow / API Server Timeout

Symptom: kubectl commands hang or return timeout. All operations are slow. Cluster feels frozen.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef api fill:#f39c12,stroke:#ba6018,color:#fff
    classDef disk fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef size fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef quorum fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff

    A["kubectl commands hang or timeout —<br/>cluster feels frozen"]:::start --> B["kubectl get --raw /healthz"]:::api
    B --> C{"API server<br/>itself healthy?"}:::api
    C -- No --> D["Check API server pod<br/>logs + restart count first —<br/>etcd may be innocent"]:::fix

    subgraph EtcdBranch["API server is healthy — the slowness is etcd itself"]
        C -- Yes --> E["Check etcd_disk_wal_fsync_duration<br/>(the write-ahead-log commit latency)"]:::disk
        E --> F{"p99 fsync<br/>&gt; 10ms?"}:::disk
        F -- Yes --> G["Disk I/O issue —<br/>check provisioned IOPS"]:::disk
        G --> H["Move etcd to<br/>faster SSD (io2 / local NVMe)"]:::fix

        F -- No --> I["etcdctl endpoint status —<br/>check DB size"]:::size
        I --> J{"DB size approaching<br/>the 2GB default quota?"}:::size
        J -- Yes --> K["etcdctl compact + etcdctl defrag<br/>(during a low-traffic window)"]:::fix
        J -- No --> L["Check etcd member health —<br/>lost quorum / split brain?"]:::quorum
    end

Commands:

# check API server health
kubectl get --raw /healthz
kubectl get --raw /readyz

# etcd health (run on etcd node or via pod)
etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

# etcd status (shows DB size)
etcdctl endpoint status --write-out=table

# defrag etcd (do during low-traffic window)
etcdctl defrag --endpoints=https://127.0.0.1:2379

Root Cause Table:

Prevention: Provision etcd on dedicated fast SSDs (io2 or local NVMe). Alert on etcd_disk_wal_fsync_duration_seconds_bucket{quantile="0.99"} > 0.01 (10ms p99). Set up automated daily compaction and defrag as a CronJob. Monitor etcd_mvcc_db_total_size_in_bytes — alert at 1.5GB (before hitting 2GB default quota). Run etcd on at least 3 nodes for quorum.

RBAC 403 — Pod Can't Call Kubernetes API

Symptom: App inside a pod gets 403 Forbidden when calling the Kubernetes API. ServiceAccount token is present but denied.

flowchart TD
    classDef start fill:#3498db,stroke:#2471a3,color:#fff
    classDef check fill:#f39c12,stroke:#ba6018,color:#fff
    classDef fix fill:#27ae60,stroke:#1e8449,color:#fff
    classDef verify fill:#8e44ad,stroke:#6c3483,color:#fff

    A["403 Forbidden in app logs —<br/>a Kubernetes API call was denied"]:::start --> B["kubectl describe pod —<br/>which ServiceAccount is attached?"]:::check
    B --> C{"Non-default SA<br/>attached?"}:::check
    C -- No --> D["Create a dedicated SA<br/>and bind a scoped Role to it —<br/>never widen the default SA"]:::fix

    subgraph RoleBranch["Dedicated SA exists — check the Role chain"]
        C -- Yes --> E["Does the Role/ClusterRole<br/>it's bound to even exist?"]:::check
        E --> F{"Role has the<br/>needed verbs/resources?"}:::check
        F -- No --> G["Add missing rules<br/>to the Role"]:::fix
        F -- Yes --> H{"RoleBinding links<br/>SA to Role?"}:::check
        H --> I{"Binding exists in<br/>the pod's own namespace?"}:::check
        I -- No --> J["Create RoleBinding /<br/>ClusterRoleBinding in the right ns"]:::fix
        I -- Yes --> K["kubectl auth can-i --as=...<br/>to verify the fix actually took"]:::verify
    end

Commands:

# test permissions as the service account
kubectl auth can-i get pods \
  --as=system:serviceaccount:default:my-sa -n default

kubectl auth can-i list secrets \
  --as=system:serviceaccount:default:my-sa -n default

# check what SA the pod uses
kubectl get pod <pod> -o jsonpath='{.spec.serviceAccountName}'

# list role bindings in namespace
kubectl get rolebinding -n <ns>
kubectl get clusterrolebinding | grep my-sa

# describe the binding
kubectl describe rolebinding <name> -n <ns>
kubectl describe clusterrolebinding <name>

Expected output snippet:

no  ← kubectl auth can-i returns "no" → RBAC is the blocker

Root Cause Table:

Prevention: Use least-privilege by default — never use cluster-admin for application service accounts. Audit existing bindings regularly: kubectl get clusterrolebinding -o json | jq '[.items[] | select(.subjects[]?.kind=="ServiceAccount")]'. Use Kyverno or OPA Gatekeeper policies to prevent broad RBAC grants in CI. Test RBAC with kubectl auth can-i --as=system:serviceaccount:ns:sa in staging.