Kubernetes Scheduler — Deep Internals

A pod that comes in without a nodeName doesn't get scheduled by magic — it moves through a strict two-phase pipeline (Filter, then Score), gets bound to whichever node wins, and only then does that node's kubelet take over and actually start it. This guide walks through that whole pipeline, what happens when no node fits, and the taint/toleration/affinity/topology-spread mechanics that feed into it.

How the Scheduler Works

The scheduler has one job: assign a nodeName to a Pod that has none. It runs a two-phase algorithm for every unscheduled pod.

flowchart TD
    WATCH["Scheduler watches API Server<br/>for pods with nodeName=''"] --> QUEUE
    QUEUE["Priority Queue<br/>(sorted by PriorityClass)"] --> FILTER
    FILTER["Phase 1: Filter (Predicates)<br/>eliminate nodes that CANNOT run the pod"] --> SCORE
    SCORE["Phase 2: Score (Priorities)<br/>rank remaining nodes 0-100"] --> BEST
    BEST["Select highest-scoring node<br/>(tie-break: random)"] --> BIND
    BIND["Bind: write nodeName to pod<br/>via API Server"] --> KUBELET
    KUBELET["kubelet on that node<br/>watches for its pods, starts container"]

Phase 1: Filter Plugins

Every filter plugin runs for every node. A node is eliminated if any filter returns false.

graph LR
    NODE["Candidate Node"] --> F1["NodeUnschedulable<br/>cordon check"]
    F1 --> F2["NodeResourcesFit<br/>enough CPU/mem?"]
    F2 --> F3["NodeAffinity<br/>nodeSelector / affinity rules"]
    F3 --> F4["TaintToleration<br/>pod tolerates node taints?"]
    F4 --> F5["PodTopologySpread<br/>spread constraints"]
    F5 --> F6["VolumeBinding<br/>PVC can be bound here?"]
    F6 --> PASS["Node passes --> enters Score phase"]
    F1 & F2 & F3 & F4 & F5 & F6 -->|"any fail"| REJECT["Node eliminated"]

Key filter plugins:

Dynamic Resource Allocation (DRA) doesn't add a row to the filter-plugin table above — it's a newer, separate scheduler extension point, the DynamicResources plugin, that sits alongside Filter/Score rather than inside either phase. That's because it isn't doing a scalar Allocatable - requested check the way NodeResourcesFit does; it resolves ResourceClaim/DeviceClass binding instead, matching structured device attributes rather than comparing a single number against capacity. See ai-infra/gpu-scheduling.md for the full treatment of DeviceClass, ResourceClaim, and the capability gap it closes over the classic GPU extended-resource model.

Phase 2: Score Plugins

Remaining nodes are scored 0-100 by each plugin. Final score = weighted sum.

graph LR
    NODES["Filtered nodes<br/>[node-1, node-2, node-3]"] --> S1["LeastAllocated<br/>prefer node with most free CPU/mem"]
    S1 --> S2["NodeAffinity<br/>preferred rules add score"]
    S2 --> S3["InterPodAffinity<br/>co-locate with preferred pods"]
    S3 --> S4["ImageLocality<br/>node already has the image? +score"]
    S4 --> FINAL["Final scores:<br/>node-1: 72<br/>node-2: 85 ← winner<br/>node-3: 61"]

What Happens When No Node Passes Filter

sequenceDiagram
    participant SCHED as Scheduler
    participant API as API Server
    participant POD as Pod

    SCHED->>SCHED: Filter: 0 nodes pass
    SCHED->>API: add Event to pod:<br/>"FailedScheduling: 0/3 nodes available:<br/>3 Insufficient cpu"
    Note over POD: Pod stays Pending indefinitely
    Note over POD: Scheduler retries every ~1s (backoff)

    Note over SCHED: If Cluster Autoscaler present:
    SCHED->>API: Node expander watches Pending pods
    API->>CA: new EC2 node provisioned
    CA->>API: node registers as Ready
    SCHED->>SCHED: retry scheduling --> node passes Filter
    SCHED->>API: Bind pod to new node

FailedScheduling events — what they mean

kubectl describe pod <pod> -n <namespace>
# Events:
#   Warning  FailedScheduling  0/3 nodes available:
#     3 Insufficient cpu                    → raise CPU requests or add nodes
#     3 node(s) had untolerated taint       → add toleration to pod
#     3 node(s) didn't match node affinity  → fix nodeSelector/affinity
#     1 pod has unbound PVC                 → PVC not bound, check StorageClass
#     3 node(s) didn't match topology       → spread constraints impossible

Full Flow: Pod Scheduled to a Node

sequenceDiagram
    participant USER as kubectl apply
    participant API as API Server (etcd)
    participant SCHED as kube-scheduler
    participant KUBELET as kubelet (node)
    participant CRI as containerd (CRI)
    participant CNI as CNI plugin
    participant POD as Pod

    USER->>API: POST /pods (Pod spec, nodeName='')
    API->>API: store in etcd, status=Pending

    SCHED->>API: watch: new pod with no nodeName
    SCHED->>SCHED: Filter + Score --> node-2 wins
    SCHED->>API: POST /pods/binding --> nodeName=node-2

    API->>KUBELET: kubelet on node-2 watches its pods
    KUBELET->>CRI: RunPodSandbox (create pause container)
    CRI->>CNI: ADD (setup network namespace, assign IP)
    CNI-->>CRI: pod IP = 10.0.1.15
    KUBELET->>CRI: PullImage (if not cached)
    KUBELET->>CRI: CreateContainer + StartContainer
    KUBELET->>API: pod status = Running, podIP = 10.0.1.15

    API->>ENDPOINT: EndpointSlice updated with new pod IP
    API->>KPROXY: kube-proxy updates iptables rules
    Note over POD: traffic now routes to this pod

Same sequence, one step at a time:

Try It Yourself: Live Filter + Score

Four nodes, 8 CPU each, all empty. Add a pod with a CPU request (leave it blank for a random 1-4) and watch the same two phases from above run for real: Filter drops any node that doesn't have enough free CPU, then Score ranks whatever's left with LeastAllocated — score = (capacity - used) / capacity, highest free-fraction wins, ties broken by lowest node index instead of the real scheduler's random tie-break so this demo stays reproducible. Push a pod too big for every remaining node and it lands in Pending instead of blocking, exactly like the FailedScheduling case above. Removing a bound pod frees its capacity immediately, but a pod already sitting in Pending is not auto-rescheduled — the real scheduler only retries on its own trigger, not the instant capacity opens up.

Taints, Tolerations, and Affinity

Taints — repel pods from nodes

# Taint a node (no GPU pods without toleration)
kubectl taint node gpu-node-1 nvidia.com/gpu=present:NoSchedule
#                             key=value:effect
# Effects: NoSchedule | PreferNoSchedule | NoExecute

The three effects aren't interchangeable severity levels of the same thing — flip through what each actually does:

Tolerations — allow pods onto tainted nodes

spec:
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Node Affinity — attract pods to nodes

spec:
  affinity:
    nodeAffinity:
      # Hard rule: pod MUST land on a node with this label
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a", "us-east-1b"]

      # Soft rule: prefer nodes with this label, but not required
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values: ["m5.2xlarge"]

These two blocks behave like the Filter/Score split from earlier — one is a hard eligibility rule, the other is only a scoring hint:

Topology Spread Constraints — spread across zones

spec:
  topologySpreadConstraints:
  - maxSkew: 1                          # max diff in pod count between zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule   # or ScheduleAnyway
    labelSelector:
      matchLabels:
        app: api