Node Shutdown and Eviction

How Kubernetes tears a node down — gracefully via kubelet's own inhibitor lock, in a controlled fashion via kubectl drain, or the messy way when a node just disappears — and how pods get evicted, in what order, in each case.

0/0 checks

1. Graceful Node Shutdown (kubelet-initiated)

When a node receives a shutdown signal (systemd shutdown.target), kubelet intercepts it via a systemd inhibitor lock and evicts pods before allowing the OS to proceed.

sequenceDiagram
    participant SD as systemd
    participant K as kubelet
    participant A as API Server
    participant P as Pods

    Note over K: kubelet registers inhibitor lock on startup
    SD->>K: Shutdown signal (reboot/poweroff)
    K->>K: Acquire inhibitor lock (blocks actual shutdown)
    K->>A: Taint node: node.kubernetes.io/not-ready:NoExecute
    Note over K: Phase 1: evict non-critical pods
    K->>P: SIGTERM --> wait grace period --> SIGKILL
    Note over K: Phase 2: evict critical pods
    K->>P: SIGTERM --> wait critical grace period --> SIGKILL
    K->>K: Release inhibitor lock
    SD->>SD: Shutdown proceeds

kubelet configuration:

# /var/lib/kubelet/config.yaml
shutdownGracePeriod: 30s              # total time for all pods to terminate
shutdownGracePeriodCriticalPods: 10s  # of the above, reserved for critical pods

Timeline with defaults:

t=0s:  Shutdown signal received
t=0s:  kubelet acquires inhibitor, begins eviction
t=0s:  Non-critical pods get SIGTERM (20s window = 30s - 10s)
t=20s: Non-critical pods get SIGKILL (if still running)
t=20s: Critical pods get SIGTERM (10s window)
t=30s: Critical pods get SIGKILL
t=30s: Inhibitor released, OS shuts down

Step through the same sequence one phase at a time:

1. Shutdown signal received (t=0s). systemd fires shutdown.target. kubelet, which registered a systemd inhibitor lock on startup, intercepts it immediately — that lock blocks the actual OS shutdown from proceeding until kubelet releases it.

2. Node tainted (t=0s). kubelet taints the node node.kubernetes.io/not-ready:NoExecute and starts evicting pods itself, in priority order.

3. Phase 1 — non-critical pods (t=0s → t=20s). SIGTERM goes out to every non-critical pod, then SIGKILL to anything still running. This phase gets whatever's left after critical pods are carved out: shutdownGracePeriod - shutdownGracePeriodCriticalPods — 20s with the defaults above.

4. Phase 2 — critical pods (t=20s → t=30s). Same SIGTERM → wait → SIGKILL pattern, but now only for critical pods (PriorityClass ≥ 2000000000), and only within the reserved shutdownGracePeriodCriticalPods window — 10s.

5. Inhibitor released, shutdown proceeds (t=30s). Only once both phases finish — or the total grace period runs out, whichever comes first — does kubelet release the inhibitor lock and let systemd continue the actual shutdown.

# Check kubelet config for shutdown settings
ssh <node>
cat /var/lib/kubelet/config.yaml | grep -i shutdown

# Verify inhibitor lock is registered
systemd-inhibit --list | grep kubelet

# Watch eviction events during shutdown
kubectl get events --field-selector reason=Evicted -A -w

With shutdownGracePeriod: 30s and shutdownGracePeriodCriticalPods: 10s, is the 10s given to critical pods in addition to the 30s, or part of it?

Part of it. shutdownGracePeriodCriticalPods is carved out of the total shutdownGracePeriod, not added on top — with these defaults, non-critical pods get the first 20s (t=0s–t=20s) and critical pods get the remaining 10s (t=20s–t=30s), for 30s total, not 40s.

2. Pod Eviction Ordering

During node shutdown (and kubelet-initiated eviction), pods are evicted in priority order:

flowchart TD
    BE["BestEffort pods<br/>(no requests/limits)<br/>evicted first"] --> BURST
    BURST["Burstable pods<br/>(requests set, usage > requests)<br/>evicted second"] --> GUAR
    GUAR["Guaranteed pods<br/>(requests == limits)<br/>evicted third"] --> CRIT
    CRIT["Critical pods<br/>(PriorityClass >= 2000000000)<br/>evicted last"]

    style BE fill:#e74c3c,color:#fff
    style BURST fill:#e67e22,color:#fff
    style GUAR fill:#27ae60,color:#fff
    style CRIT fill:#2980b9,color:#fff

PriorityClass determines "critical":

PriorityClass	Value	Examples
`system-node-critical`	2000001000	kubelet, kube-proxy, CNI
`system-cluster-critical`	2000000000	CoreDNS, metrics-server
(user pods)	< 1000000000	Your workloads

# Protect your pod from early eviction
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
---
spec:
  priorityClassName: high-priority

Does giving a pod requests equal to its limits (QoS class Guaranteed) protect it from eviction as well as giving it a PriorityClass ≥ 2000000000 would?

3. Node Drain (Voluntary Eviction)

kubectl drain is the controlled, human-initiated version of node shutdown. Used for maintenance, node upgrades, and cluster autoscaler scale-down.

flowchart TD
    DRAIN["kubectl drain NODE_NAME"] --> CORDON
    CORDON["kubectl cordon NODE_NAME<br/>node.spec.unschedulable = true<br/>No new pods scheduled here"] --> EVICT
    EVICT["Evict each pod via Eviction API<br/>(respects PodDisruptionBudget)"] --> WAIT
    WAIT["Wait for each pod to terminate<br/>(terminationGracePeriodSeconds)"] --> DONE
    DONE["Node ready for maintenance"]

    EVICT -->|"PDB would be violated"| BLOCK["Drain blocked<br/>until PDB allows eviction"]
    BLOCK --> EVICT

Step through it:

1. kubectl drain <node> invoked. Human- or automation-initiated — same eviction machinery as a graceful node shutdown, but controlled, and PDB-aware.

2. Node cordoned. kubectl cordon sets node.spec.unschedulable = true first. No new pods land here, but nothing already running is touched yet.

3. Pods evicted via the Eviction API. Each evictable pod gets an eviction request — not a raw delete — which explicitly checks any PodDisruptionBudget covering it.

4. PDB check: proceed or block. If evicting this pod would drop its PDB below minAvailable, the request is refused, not queued. Drain keeps retrying automatically until the PDB allows it.

5. Wait for termination. Each evicted pod gets its normal terminationGracePeriodSeconds (or the drain's --grace-period override) to shut down.

6. Node ready for maintenance. Once every evictable pod is gone, drain returns. kubectl uncordon is what makes the node schedulable again afterward — drain itself never does that automatically.

# Full drain command
kubectl drain <node-name> \
  --ignore-daemonsets \        # DaemonSet pods can't be evicted (managed by DS controller)
  --delete-emptydir-data \     # evict pods using emptyDir (data will be lost)
  --timeout=5m \               # fail if drain takes longer than 5 min
  --grace-period=30            # override pod terminationGracePeriodSeconds

# Uncordon after maintenance
kubectl uncordon <node-name>

# Check what's blocking drain
kubectl get pdb -A             # list PodDisruptionBudgets
kubectl describe pdb <name>    # check minAvailable vs current ready pods

Why drain blocks:

graph LR
    PDB["PDB: minAvailable=2"] --> Ready["current ready = 2"]
    Ready -->|evicting 1 more| Violate["violates PDB"]
    Violate --> Blocked["drain blocked,<br/>retries automatically"]
    Blocked -->|replacement pod<br/>comes up elsewhere| Ready2["ready > minAvailable"]
    Ready2 --> Proceed["drain proceeds"]

    style Violate fill:#e67e22,color:#fff
    style Blocked fill:#c0392b,color:#fff
    style Proceed fill:#27ae60,color:#fff

A PDB has minAvailable=2 and there are exactly 2 ready replicas. What happens when kubectl drain tries to evict one of them?

4. Non-Graceful Node Shutdown

If a node dies abruptly (power cut, kernel panic, kill -9 kubelet), the node object stays in the cluster but kubelet is gone. Pods on that node stay in Terminating forever — the API server is waiting for kubelet to confirm deletion, which never comes.

# Pods stuck Terminating on dead node
kubectl get pods --field-selector spec.nodeName=<dead-node>

# Old manual fix (still works)
kubectl delete pod <pod> --grace-period=0 --force

# Better: delete the node object
kubectl delete node <dead-node>
# → node lifecycle controller adds not-ready/unreachable taint
# → pods get evicted after tolerationSeconds

Kubernetes 1.28+ — out-of-service taint (GA):

# Manually taint the dead node
kubectl taint node <dead-node> \
  node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

# Effect: StatefulSet pods and pods with PVCs are immediately force-deleted
# and rescheduled on healthy nodes — even without kubelet confirmation
# This was the #1 StatefulSet recovery pain point before this feature

Same underlying goal — get pods off a node that's going away — but graceful and non-graceful shutdown look nothing alike from the API server's point of view:

kubelet is alive and in control. It intercepts the shutdown via its systemd inhibitor lock, evicts pods itself in priority order (non-critical, then critical) within a bounded grace period, then releases the lock. The API server sees clean, confirmed pod terminations the entire way through — nothing is ever stuck.

kubelet is gone before it can do anything. No inhibitor lock, no orderly SIGTERM/SIGKILL phases — the node just disappears. Pods are stuck Terminating because the API server is waiting on a kubelet confirmation that will never arrive. Recovery needs a human or automation to step in: delete the node object, or apply the out-of-service taint, before anything gets rescheduled.

A node dies from a kernel panic. Why do its pods show Terminating forever instead of being cleaned up automatically?

Because kubelet — the process that would normally confirm pod deletion back to the API server — is gone. The API server marked the pods for deletion but is still waiting on that confirmation, which never comes until something intervenes manually: deleting the node object, or applying the out-of-service taint.

5. Node Lifecycle Taints

The node lifecycle controller automatically adds taints when conditions change. These taints trigger pod eviction via tolerationSeconds.

Condition	Taint key	Auto-tolerationSeconds on pods
`NotReady`	`node.kubernetes.io/not-ready`	300s (default)
`Unreachable`	`node.kubernetes.io/unreachable`	300s (default)
`MemoryPressure`	`node.kubernetes.io/memory-pressure`	—
`DiskPressure`	`node.kubernetes.io/disk-pressure`	—
`PIDPressure`	`node.kubernetes.io/pid-pressure`	—
`NetworkUnavailable`	`node.kubernetes.io/network-unavailable`	—

tolerationSeconds controls how long a pod tolerates a taint before eviction:

# Default toleration on all pods (injected by admission controller):
tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 300   # stay on not-ready node for 5 min before evicted

# For fast failover (stateless services):
tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 30    # evict in 30s

# For sticky workloads (StatefulSets, cache):
tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 600   # give node 10 min to recover before moving pod

# Check node conditions and taints
kubectl describe node <node> | grep -A10 Taints
kubectl describe node <node> | grep -A20 Conditions

# Watch node status changes
kubectl get nodes -w

# See eviction events
kubectl get events -A --field-selector reason=Evicted
kubectl get events -A | grep "Evicting\|Evicted\|Killing"

Of NotReady, Unreachable, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable, which conditions get an automatic tolerationSeconds timer that evicts pods on a delay?

6. Kubelet Eviction Thresholds

kubelet watches node resources and evicts pods before the node itself runs out.

# kubelet config: /var/lib/kubelet/config.yaml
evictionHard:
  memory.available: "200Mi"     # evict when < 200Mi free
  nodefs.available: "10%"       # evict when < 10% disk
  nodefs.inodesFree: "5%"       # evict when < 5% inodes
  imagefs.available: "15%"      # evict when < 15% image filesystem

evictionSoft:
  memory.available: "500Mi"     # warn when < 500Mi
evictionSoftGracePeriod:
  memory.available: "1m30s"     # only evict if soft threshold exceeded for 90s

evictionMinimumReclaim:
  memory.available: "100Mi"     # reclaim at least 100Mi per eviction round

Eviction order within a QoS class: pods using the most resources above their requests are evicted first.

A node's memory.available drops to 480Mi, breaching the evictionSoft threshold of 500Mi for a moment before recovering at 60s. Does kubelet evict a pod?