HPA and VPA — Deep Internals

Every major section below ends with a quick knowledge check — try to answer before revealing.

0/0 checks

HPA — Horizontal Pod Autoscaler

Every Stage Internals

sequenceDiagram
    participant APP as App Pod
    participant MS as Metrics Server
    participant HPA as HPA Controller
    participant API as API Server
    participant SCHED as Scheduler
    participant KUBELET as kubelet
    participant CA as Cluster Autoscaler

    Note over APP,MS: Every 15s (scrape interval)
    APP->>MS: expose /metrics (cAdvisor on kubelet)
    MS->>MS: aggregate CPU/memory per pod

    Note over HPA: Every 15s (--horizontal-pod-autoscaler-sync-period)
    HPA->>MS: GET /apis/metrics.k8s.io/v1beta1/namespaces/default/pods
    MS-->>HPA: [{pod: api-1, cpu: 85m}, {pod: api-2, cpu: 90m}]

    HPA->>HPA: compute desired replicas
    Note over HPA: desiredReplicas = ceil(currentReplicas × (currentMetric/targetMetric))<br/>= ceil(2 × (87.5/70)) = ceil(2.5) = 3

    HPA->>API: PATCH deployment/api spec.replicas=3
    API->>API: ReplicaSet controller sees 2 pods, wants 3
    API->>SCHED: new Pod created (Pending, no node assigned)

    SCHED->>API: GET nodes + pods (from cache)
    SCHED->>SCHED: Filter --> Score --> Bind
    SCHED->>API: Bind pod to node-2

    API->>KUBELET: kubelet on node-2 watches pod assigned to it
    KUBELET->>KUBELET: pull image, create sandbox, start container
    KUBELET->>API: pod status = Running

    Note over HPA: If no node has capacity:
    SCHED-->>API: pod stays Pending
    CA->>CA: detects Pending pod, provisions new EC2 node
    CA->>SCHED: new node joins, pod gets scheduled

Same flow, one stage at a time:

1. Metrics exposed. Every pod exposes CPU/memory via cAdvisor on its kubelet. This happens continuously, independent of HPA.

2. Metrics Server scrapes and aggregates. Every 15s (scrape interval) it collects per-pod CPU/memory and serves it at metrics.k8s.io.

3. HPA polls. Every --horizontal-pod-autoscaler-sync-period (default 15s), the HPA controller calls GET /apis/metrics.k8s.io/v1beta1/.../pods against Metrics Server.

4. HPA computes desired replicas. desiredReplicas = ceil(currentReplicas × (currentMetric/targetMetric)) — e.g. ceil(2 × (87.5/70)) = 3.

5. HPA patches the Deployment. It PATCHes spec.replicas via the API server. HPA never creates a Pod itself — it only ever changes a number on the scale target.

6. ReplicaSet controller reacts. It sees fewer running pods than spec.replicas asks for and creates the missing Pod objects — Pending, unscheduled.

7. Scheduler binds. Filter → Score → Bind assigns the new Pod to a node with enough free capacity.

8. kubelet starts it — or Cluster Autoscaler steps in. If a node was bound, kubelet pulls the image and starts the container. If Filter found no eligible node, the Pod stays Pending until Cluster Autoscaler provisions one.

Does the HPA controller create Pods directly when it scales up?

No. HPA only PATCHes spec.replicas on the scale target (a Deployment, typically). The ReplicaSet controller is what actually notices the new replica count and creates the Pod objects — HPA's job ends the moment it writes that one number.

The Math

desiredReplicas = ceil( currentReplicas × (currentValue / targetValue) )

Example: 3 pods, CPU target 70%, current average 90%
desiredReplicas = ceil(3 × 90/70) = ceil(3.86) = 4

Scale-down: 4 pods, CPU 20%
desiredReplicas = ceil(4 × 20/70) = ceil(1.14) = 2
BUT: scale-down waits stabilizationWindowSeconds (default 300s)
to avoid flapping

The formula says 4 pods at 20% CPU should scale down to 2 right now. Does HPA apply that immediately?

No — scale-down is gated behind stabilizationWindowSeconds (300s by default). Instead of reacting to the single latest computation, HPA takes the highest recommended replica count seen over that whole window before actually scaling down. Scale-up has no such default delay; scale-down does, specifically to avoid flapping on a brief dip.

HPA QoS Classes and Behaviour

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # scale up immediately
      policies:
      - type: Pods
        value: 4                         # add at most 4 pods per 60s
        periodSeconds: 60
      - type: Percent
        value: 100                       # or double pods per 60s
        periodSeconds: 60
      selectPolicy: Max                  # use whichever is larger

    scaleDown:
      stabilizationWindowSeconds: 300    # wait 5min before scaling down
      policies:
      - type: Pods
        value: 1                         # remove at most 1 pod per 60s
        periodSeconds: 60

scaleUp.selectPolicy: Max is set with two policies: add 4 pods/60s, or double pods/60s. Does Max pick whichever policy produces the larger replica count, or the smaller?

The larger. selectPolicy: Max applies whichever policy allows the biggest change — here, whichever of "+4 pods" or "double" adds more replicas wins. Min would do the opposite, capping growth to the smaller of the two.

What Happens When No Node is Available

flowchart TD
    HPA["HPA: scale to 5 replicas"] --> RS["ReplicaSet creates<br/>new Pod object"]
    RS --> SCHED["Scheduler: Filter phase<br/>no node passes all filters"]
    SCHED --> PENDING["Pod status: Pending<br/>Event: FailedScheduling<br/>'Insufficient cpu'"]
    PENDING --> CA["Cluster Autoscaler<br/>watches Pending pods"]
    CA --> ASG["Expand ASG / Node Group<br/>provision new EC2 node"]
    ASG --> NODE["New node Ready<br/>registers with API Server"]
    NODE --> SCHED2["Scheduler retries<br/>pod gets bound to new node"]
    SCHED2 --> KUBELET["kubelet: pull image<br/>start container"]
    KUBELET --> RUNNING["Pod: Running"]

HPA scales a Deployment to 5 replicas, but no node has enough free CPU to schedule the new Pod. What happens to HPA's desired replica count?

Nothing reverts. The desired count stays at 5 — the new Pod just sits Pending with a FailedScheduling / "Insufficient cpu" event until Cluster Autoscaler notices it and provisions a new node. HPA doesn't retry or back off; it already did its job by writing the replica count.

Debugging HPA

# See current HPA state
kubectl get hpa -n <namespace>
# NAME   REFERENCE         TARGETS         MINPODS  MAXPODS  REPLICAS
# api    Deployment/api    85%/70%         2        20       4

# Describe for events and conditions
kubectl describe hpa api -n <namespace>
# Conditions:
#   AbleToScale    True   SucceededGetScale
#   ScalingActive  True   ValidMetricFound
#   ScalingLimited False  DesiredWithinRange

# Check metrics server is working
kubectl top pods -n <namespace>
# If this fails → HPA will show <unknown>

# Events that matter
kubectl get events -n <namespace> | grep HPA
# SuccessfulRescale: New size: 4; reason: cpu resource utilization above target

VPA — Vertical Pod Autoscaler

Why VPA?

HPA adds more pods. VPA makes each pod bigger (more CPU/memory). Use when:

Singleton workloads — only one instance can run (leader-elected, stateful, CronJob)
Memory-bound workloads — memory doesn't decrease with more replicas (ML model loaded in memory)
Right-sizing — you don't know the right requests/limits; start with VPA in Off mode for recommendations

Why doesn't HPA help a singleton, memory-bound workload (e.g. a leader-elected service holding a large in-memory model)?

VPA Architecture

graph TD
    VPA_ADM["VPA Admission Controller<br/>(MutatingWebhook)"] -->|"inject requests on pod create"| POD["Pod<br/>resources patched at creation"]
    VPA_REC["VPA Recommender<br/>(reads metrics history)"] -->|"update VPA object"| VPA_OBJ["VPA Object<br/>spec.updatePolicy.updateMode"]
    VPA_UPD["VPA Updater<br/>(evicts pods to apply)"] -->|"evict pod if limits too low"| POD
    METRICS["Metrics Server / Prometheus"] -->|"historical CPU/mem"| VPA_REC

Three components:

Recommender — watches pod resource usage history, computes lowerBound, target, upperBound
Updater — evicts pods that are too far from target (so Admission Controller can inject new values on restart)
Admission Controller — patches resources on pod creation (Mutating webhook)

The three components as one cycle:

1. Metrics collected. Metrics Server / Prometheus records each container's historical CPU and memory usage over time.

2. Recommender computes bounds. It watches that history and writes lowerBound, target, and upperBound per container into the VPA object's status.

3. Updater compares to target. If the running pod's current resources are too far from the recommended target, the Updater evicts it — only in Recreate/Auto mode.

4. Admission Controller patches on restart. When the evicted pod's replacement is created, the MutatingWebhook injects the new resources.cpu/memory before it starts. The running pod itself is never edited in place.

Which VPA component actually restarts a running pod, and which one sets its new CPU/memory values?

The Updater triggers the restart, by evicting the pod once its resources are too far from the Recommender's target. The Admission Controller is what actually sets the new values, by patching resources on the replacement pod at creation time — the Updater itself never edits a pod's resources, it only evicts.

VPA Update Modes

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Auto"    # Off | Initial | Recreate | Auto
  resourcePolicy:
    containerPolicies:
    - containerName: api
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 4
        memory: 4Gi
      controlledResources: ["cpu", "memory"]

Mode	What it does
`Off`	Recommendations only — no changes applied. Use first to baseline.
`Initial`	Sets requests at pod creation, never touches running pods
`Recreate`	Evicts pods to apply new recommendations (causes restart)
`Auto`	Same as Recreate today; future: in-place update

In Off mode, does the VPA Recommender still run and calculate target CPU/memory?

Yes. Off only stops changes being applied — the Recommender keeps computing lower/target/upper bounds the whole time. That's exactly why Off is the recommended first step: you get real recommendations to eyeball before ever letting Recreate/Auto touch a running pod.

VPA for Singleton Applications

sequenceDiagram
    participant VPA_R as VPA Recommender
    participant VPA_U as VPA Updater
    participant POD as Singleton Pod
    participant API as API Server

    Note over VPA_R: monitors CPU/mem over time
    VPA_R->>API: update VPA status: target.cpu=500m target.memory=1Gi

    Note over VPA_U: pod is using 200m CPU but limit is 100m --> throttled
    VPA_U->>POD: evict (SIGTERM)
    Note over POD: pod restarts
    VPA_ADM->>POD: on admission: inject cpu=500m memory=1Gi
    POD->>API: running with correct resources

    Note over POD: ⚠️ Singleton = downtime during eviction
    Note over POD: Mitigation: PodDisruptionBudget minAvailable=0<br/>or use VPA updateMode=Initial

Singleton + VPA caveat: Every Updater eviction = restart = downtime for singleton. Mitigations:

Use updateMode: Initial — recommendations applied at next natural restart only
Use updateMode: Off + manually apply recommendations during maintenance window
Use minAvailable: 0 in PDB to allow the eviction but schedule the restart yourself

For a singleton pod, what's the practical effect every time the VPA Updater evicts it to apply a new recommendation?

Downtime. A normal Deployment has other replicas to absorb traffic during an eviction; a singleton has none, so every Updater-triggered eviction is a real outage window until the pod restarts and the Admission Controller re-injects its resources. That's why singleton + VPA calls for updateMode: Initial/Off or a PDB with minAvailable: 0, not plain Auto.

Checking VPA Recommendations

kubectl describe vpa api-vpa
# Status:
#   Recommendation:
#     Container Recommendations:
#       Container Name: api
#       Lower Bound:
#         Cpu:     100m
#         Memory:  256Mi
#       Target:                  ← use this for requests
#         Cpu:     400m
#         Memory:  768Mi
#       Uncapped Target:
#         Cpu:     400m
#         Memory:  768Mi
#       Upper Bound:             ← use this for limits
#         Cpu:     2
#         Memory:  2Gi

HPA + VPA: Can You Use Both?

Not on the same metric. Whether combining them works or fights itself comes down entirely to what each one is watching. Flip between the two states:

Both target CPU. VPA increases pod CPU → pods use less CPU → HPA reads that as under target and scales in (fewer pods) → more load lands on each surviving pod → VPA increases CPU again → loop. Two controllers reacting to the same signal from opposite directions, each one's action becoming the other's next trigger.

Different signals, or one side passive. HPA on a custom metric (RPS, queue depth) + VPA on CPU/memory — they're no longer watching the same number, so one's correction doesn't retrigger the other. Or: VPA in Off mode (recommendations only) + HPA on CPU — VPA never actually changes anything, you apply its recommendations manually, so there's nothing for HPA to react to.

HPA and VPA are both configured against CPU utilization on the same Deployment. Why does this create a feedback loop instead of just working extra well?