SRE & Debugging Reference

Runbooks, debugging scenarios, and SRE concepts for Kubernetes, Linux, and AWS.

Index

K8s Debugging — Quick Triage

pod not Running?
  Pending          → kubectl describe pod → FailedScheduling events
  CrashLoopBackOff → kubectl logs --previous + exit code
  ImagePullBackOff → kubectl describe pod → registry/auth/tag
  Terminating      → kubectl get pod -o json | jq .metadata.finalizers

pod Running but broken?
  Requests failing → kubectl exec -- curl localhost:PORT, check logs
  DNS failing      → kubectl exec -- nslookup my-svc
  Can't reach pod  → check NetworkPolicy, check Service endpoints
  High latency     → kubectl top pod, check CPU throttling

node issues?
  NotReady         → kubectl describe node → Conditions + Events
  Disk pressure    → kubectl debug node, df -h on node
  etcd slow        → etcdctl endpoint health, etcdctl defrag

Linux Debugging — Quick Triage

high load?
  CPU high + nothing in top → perf top (kernel threads / softirq / steal)
  CPU idle + high load      → iostat -x (iowait), iotop (culprit process)

process issues?
  zombies          → ps -eo pid,ppid,stat | awk '$3~/Z/' → kill parent
  disappeared      → dmesg | grep -i oom (OOM killer)
  fd exhaustion    → lsof -p PID | wc -l, ulimit -n

disk issues?
  no space (df fine) → df -i (inodes exhausted)
  slow writes        → iostat -x (await > 100ms), iotop
  nfs hang           → umount -l, check server reachability

crashes?
  unexpected reboot  → journalctl -b -1 | tail -50
  kernel panic       → dmesg | grep -i panic, /var/crash/vmcore