Load Balancers — Deep Dive

Beginner to advanced reference covering L4/L7, algorithms, health checks, TLS, AWS ALB/NLB, GCP, nginx, HAProxy, and common failure modes.

1. What a Load Balancer Does

A load balancer sits between clients and backend servers and provides:

• Traffic distribution — spread requests across a pool of backends
• Health checking — remove unhealthy backends from rotation automatically
• TLS termination — decrypt HTTPS at the LB so backends speak plain HTTP
• Backend topology hiding — clients see one VIP; backend IPs are never exposed
• Connection management — keep persistent upstream pools, draining on deploys

2. L4 vs L7

NAT vs Proxy Model

3. L4 vs L7 — Mermaid Diagram

graph TD
    classDef client fill:#34495e,stroke:#212f3c,color:#fff
    classDef lb4 fill:#16a085,stroke:#117a65,color:#fff
    classDef lb7 fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef backend fill:#2980b9,stroke:#1f618d,color:#fff

    subgraph L4_LB["L4 LB — TCP Passthrough with DNAT (one connection end-to-end)"]
        C1["Client"]:::client -->|"TCP SYN<br/>dst 10.0.0.1:443"| LB4["L4 Load Balancer<br/>never terminates the TCP session"]:::lb4
        LB4 -->|"DNAT: dst rewritten<br/>to 10.0.1.2:443"| B1["Backend 1"]:::backend
        LB4 -->|"DNAT: dst rewritten<br/>to 10.0.1.3:443"| B2["Backend 2"]:::backend
    end

    subgraph L7_LB["L7 LB — Full HTTP Proxy (two decoupled connections)"]
        C2["Client"]:::client -->|"TCP conn 1: HTTPS"| LB7["L7 Load Balancer<br/>terminates TLS, parses HTTP<br/>headers / path / host"]:::lb7
        LB7 -->|"TCP conn 2: HTTP<br/>to chosen backend"| B3["Backend A"]:::backend
        LB7 -->|"TCP conn 2: HTTP<br/>to chosen backend"| B4["Backend B"]:::backend
    end

4. Load Balancing Algorithms

Try It Yourself: Live Algorithm Routing

The tabs above describe each algorithm in the abstract. This is live: add backends with whatever weights you want, pick an algorithm, and fire requests one at a time (or many, fast). Each backend's corner badge is a running connection count that increments the moment it's routed to. Kill a backend mid-run and watch every algorithm route around it immediately — then revive it and watch it rejoin rotation.

5. Health Checks

Active Health Checks

The LB probes backends on a schedule, independent of real traffic.

Key parameters:

• interval — how often to probe (e.g., 10s)
• timeout — how long to wait for a response (e.g., 5s)
• unhealthy threshold — consecutive failures before marking down (e.g., 2)
• healthy threshold — consecutive successes before marking up (e.g., 3)
• grace period — time after startup before health checks are enforced (avoids killing pods during JVM warm-up)

Passive Health Checks (Circuit Breaker)

Watch real traffic error rates. If backend returns 5xx above threshold, eject it from the pool for a cooldown window.

Use when: active probing misses transient errors (a backend is up but throwing errors for a specific endpoint).

Envoy / Istio call this outlier detection: consecutive 5xx count → eject for base_ejection_time (doubles each ejection).

6. Sticky Sessions

Cookie-Based (Preferred)

LB-generated cookie: LB inserts a cookie (e.g., AWSALB) on first response, encodes the target backend. On subsequent requests LB reads cookie → routes to same backend.

App-generated cookie: App sets a session cookie; LB reads it and hashes it to a backend.

IP Hash

See §4. Unreliable behind NAT or IPv6 CGNAT.

Why Sticky Sessions Are Problematic

• Uneven load: one sticky client can hammer one backend (e.g., long-running WebSocket or batch job)
• Failover loses session: if the sticky backend dies, the session is gone — unless app replicates session to Redis/DB
• Defeats autoscaling: new backends receive no traffic from existing sticky clients
• Recommendation: prefer stateless services + external session store (Redis/DynamoDB) over sticky sessions

7. Connection Draining / Deregistration Delay

Why It Exists

When a backend is removed from the pool (deploy, scale-down, health failure), in-flight requests are mid-stream. Killing the connection immediately → client gets a 502/reset.

How It Works

sequenceDiagram
    participant Client as Existing Client
    participant NewClient as New Client
    participant LB
    participant Backend as Backend (draining)
    participant Healthy as Healthy Backend

    Note over Backend: Deploy / scale-down / health failure<br/>triggers removal from the pool
    LB->>LB: Mark Backend as draining —<br/>stop routing NEW requests to it
    NewClient->>LB: New request arrives
    LB->>Healthy: Routed to a healthy backend instead
    Healthy-->>LB: Response
    LB-->>NewClient: Response

    Client->>LB: In-flight request (already routed before draining)
    LB->>Backend: Forward in-flight request — draining backends still finish work already in flight
    Backend-->>LB: Response
    LB-->>Client: Response

    Note over LB,Backend: Deregistration delay window<br/>(default 300s on ALB, often tuned to 30s)
    LB->>Backend: Timeout expires — forcefully close any remaining connections

Tune to: slightly above your p99 request duration. 30s is usually enough for APIs; leave 300s for long-running uploads.

8. TLS Termination

graph LR
    classDef enc fill:#c0392b,stroke:#922b21,color:#fff
    classDef plain fill:#7f8c8d,stroke:#616a6b,color:#fff
    Client["Client"]:::enc -->|"HTTPS - encrypted"| LB["LB - terminates TLS"]:::enc
    LB -->|"HTTP - plaintext"| Backend["Backend"]:::plain

graph LR
    classDef enc fill:#c0392b,stroke:#922b21,color:#fff
    Client["Client"]:::enc -->|"HTTPS - encrypted"| LB["L4 LB - SNI routing only, never decrypts"]:::enc
    LB -->|"HTTPS - still encrypted"| Backend["Backend - holds the cert"]:::enc

graph LR
    classDef enc fill:#c0392b,stroke:#922b21,color:#fff
    Client["Client"]:::enc -->|"HTTPS - encrypted"| LB["LB - terminates, inspects, re-encrypts"]:::enc
    LB -->|"HTTPS - re-encrypted"| Backend["Backend"]:::enc

graph LR
    classDef svc fill:#8e44ad,stroke:#6c3483,color:#fff
    A["Service A"]:::svc -->|"mTLS - both sides present certs"| SA["Istio sidecar"]:::svc
    SA -->|"mTLS"| SB["Service B sidecar"]:::svc

9. AWS ALB Deep Dive

Architecture

graph LR
    classDef client fill:#34495e,stroke:#212f3c,color:#fff
    classDef alb fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef tg fill:#2980b9,stroke:#1f618d,color:#fff

    Client["Client"]:::client -->|"HTTPS"| ALB["ALB Listener :443<br/>ordered rule evaluation"]:::alb

    subgraph RULES["Listener rules"]
        ALB -->|"Rule: path /api/*"| TG1["Target Group: API pods"]:::tg
        ALB -->|"Rule: path /static/*"| TG2["Target Group: S3 / Lambda"]:::tg
        ALB -->|"Rule: host app2.example.com"| TG3["Target Group: App2"]:::tg
        ALB -->|"Default rule (no match above)"| TG4["Target Group: Default"]:::tg
    end

Listeners, Rules, Target Groups

• Listener: protocol + port (HTTP:80, HTTPS:443). Each listener has ordered rules.
• Rule conditions: path pattern, host header, HTTP header, HTTP method, query string, source IP
• Rule actions: forward, redirect, fixed response, authenticate (Cognito/OIDC), weighted forward (canary)
• Target group: EC2 instances, IP addresses (pods), Lambda functions, ALB (nested)

Content-Based Routing Examples

Rule 1: host = api.example.com AND path = /v2/* → TG_v2 (canary 10%) + TG_v1 (90%)
Rule 2: path = /health                           → fixed 200
Rule 3: header X-Beta = true                     → TG_beta
Default:                                         → TG_main

Protocol Support

Access Logs

Enable per-listener to S3. Fields include: client IP, timestamp, target IP, request processing time, target processing time, response time, status codes, SSL cipher, user-agent, request ID.

10. AWS NLB Deep Dive

Key Properties

• Layer 4 — TCP, UDP, TLS protocols
• Static IPs per AZ — one Elastic IP per AZ. Safe to whitelist in firewalls (ALB IPs change).
• Preserve client IP — unlike ALB (which replaces src IP), NLB passes the real client IP to the backend (security group must allow it)
• Ultra-low latency — no HTTP parsing overhead; millions of requests/second
• TLS offload — NLB can terminate TLS (similar to ALB), with ACM certificates
• UDP — useful for DNS, syslog, game servers
• PrivateLink — expose a service to another VPC/account via NLB without VPC peering

NLB vs ALB Decision

graph TD
    classDef q fill:#34495e,stroke:#212f3c,color:#fff
    classDef alb fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef nlb fill:#16a085,stroke:#117a65,color:#fff

    Q1{"Need content-based routing?"}:::q -->|"Yes"| ALB["Use ALB"]:::alb
    Q2{"Need a static IP?"}:::q -->|"Yes"| NLB["Use NLB"]:::nlb
    Q3{"Need UDP?"}:::q -->|"Yes"| NLB
    Q4{"Need a PrivateLink endpoint?"}:::q -->|"Yes"| NLB
    Q5{"Need WAF?"}:::q -->|"Yes"| ALB
    Q6{"Need a Lambda target?"}:::q -->|"Yes"| ALB
    Q7{"Is raw TCP performance the priority?"}:::q -->|"Yes"| NLB

11. LCU and NLCU — Capacity Units and Pricing

AWS bills ALB and NLB on two axes: a flat hourly charge plus a capacity-unit charge based on actual usage. Understanding the capacity unit is essential for cost forecasting and for diagnosing throttling under load. Elastic Load Balancing does not bill you the sum of all dimensions — it bills you on the single highest dimension consumed in each hour.

11.1 ALB — Load Balancer Capacity Unit (LCU)

An LCU measures the traffic an ALB processes across four independent dimensions. Each hour, AWS computes how many LCUs you consumed on each dimension, takes the maximum, and bills that.

Billed LCUs for the hour = MAX(
    new_connections_dim,
    active_connections_dim,
    processed_bytes_dim,
    rule_evaluations_dim
)

Rule evaluations is the subtle one. The first 10 rule evaluations per request are free. If a request matches after evaluating 15 rules, only 5 count. The dimension value is:

rule_eval_LCU = (request_rate/sec × max(0, rules_evaluated − 10)) / 1000

So a rule-heavy listener (deep rule chains, many host/path conditions) can make rule evaluations — not bytes — the dominant cost driver.

Worked ALB Example

An API service over one hour:

• 1,000 new connections/sec
• 60,000 active connections (sampled/min)
• 5 GB/hour processed
• 20 rules evaluated per request, 1,000 requests/sec

New connections:    1,000 / 25       = 40.0  LCU
Active connections: 60,000 / 3,000   = 20.0  LCU
Processed bytes:    5 GB / 1 GB      =  5.0  LCU
Rule evaluations:   (1,000 × (20−10)) / 1,000 = 10.0 LCU

Billed = MAX(40, 20, 5, 10) = 40 LCU  → new connections dominates

At the us-east-1 rate of $0.008/LCU-hour:

LCU cost  = 40 LCU × $0.008              = $0.32/hour
Hourly LB = $0.0225/hour (ALB base)      = $0.0225/hour
Total     ≈ $0.34/hour  ≈ $248/month

Takeaway: new-connection rate dominated here. Enabling HTTP keep-alive so clients reuse connections would collapse the new-connection dimension and cut the bill dramatically.

11.2 NLB — Network Load Balancer Capacity Unit (NLCU)

NLB uses NLCU, with dimensions that differ by protocol (TCP vs UDP vs TLS). Only three dimensions, and again you're billed on the max.

Protocol-specific rates matter:

TLS on NLB is expensive — the TLS dimension gives you only 50 new connections/sec per NLCU (vs 800 for raw TCP), because the NLB does the TLS handshake termination. High-churn TLS connections on an NLB burn NLCUs fast.

Worked NLB Example (TLS)

A TLS service: 500 new TLS connections/sec, 30,000 active flows, 8 GB/hour.

New TLS connections: 500 / 50        = 10.0  NLCU
Active flows:        30,000 / 3,000  = 10.0  NLCU
Processed bytes:     8 GB / 1 GB     =  8.0  NLCU

Billed = MAX(10, 10, 8) = 10 NLCU

At $0.006/NLCU-hour:

NLCU cost = 10 × $0.006             = $0.06/hour
Hourly LB = $0.0225/hour (NLB base) = $0.0225/hour
Total     ≈ $0.083/hour ≈ $60/month

11.3 ALB LCU vs NLB NLCU — Side by Side

Why NLB is cheaper at scale for raw TCP: 1 NLCU absorbs 800 new connections/sec vs 25 for an LCU — a 32× difference on the connection dimension. For high-throughput TCP with long-lived connections, NLB's capacity-unit math is far more favorable.

11.4 Capacity Planning and the Max-Dimension Trap

The single most common costing mistake is optimizing the wrong dimension. Always identify which dimension is your binding constraint:

Workload profile                    Likely dominant dimension
────────────────────────────────────────────────────────────
Chatty API, no keep-alive          New connections   → enable keep-alive
WebSockets / long-poll             Active connections → size for concurrency
Large file downloads / streaming   Processed bytes    → consider CloudFront offload
Complex routing (many rules)       Rule evaluations   → flatten rule chains, use host-based
Bulk TLS handshakes on NLB         New TLS flows      → move TLS term to ALB or reuse conns

Diagnose with CloudWatch — each dimension has a metric:

# ALB consumed LCUs (and per-dimension breakdown)
aws cloudwatch get-metric-statistics \
  --namespace AWS/ApplicationELB \
  --metric-name ConsumedLCUs \
  --dimensions Name=LoadBalancer,Value=app/my-alb/50dc6c495c0c9188 \
  --start-time 2024-01-15T00:00:00Z \
  --end-time 2024-01-15T01:00:00Z \
  --period 3600 --statistics Maximum

# Per-dimension ALB metrics to find the binding constraint:
#   NewConnectionCount, ActiveConnectionCount,
#   ProcessedBytes, RuleEvaluations

# NLB consumed capacity
aws cloudwatch get-metric-statistics \
  --namespace AWS/NetworkELB \
  --metric-name ConsumedLCUs \
  --dimensions Name=LoadBalancer,Value=net/my-nlb/... \
  --period 3600 --statistics Maximum

11.5 Pre-Warming and Scaling Behavior

ELB scales its own capacity gradually — it is not instant. When traffic jumps faster than the LB can add capacity, you see 503s (ALB) or connection failures (NLB).

• ALB/NLB scale up over minutes, targeting your observed traffic trend
• A sudden 10× spike (flash sale, viral event, load test) can outrun the scaling
• AWS no longer offers self-service pre-warming; for known spikes you open a support case (Business/Enterprise support) to request pre-warming, or ramp load gradually
• NLB scales faster than ALB for connection spikes because it does no L7 parsing — another reason to front extreme TCP bursts with NLB
• Load-test realistically: ramp up, don't slam from 0 to peak, or you'll measure the LB's scaling curve rather than your app

Traffic pattern that triggers 503s:
   requests/sec
        │           ╱│  ← instant 10× spike outruns LB scaling
        │          ╱ │     → 503 Service Unavailable
        │       ___╱  │
        │   ___╱      │  ← LB capacity (lags behind)
        └──────────────── time

Safe pattern:
        │        ____╱  ← gradual ramp, LB keeps pace
        │    ___╱
        └──────────────── time

11.6 Cost Optimization Checklist

• Enable HTTP keep-alive — collapses the new-connections dimension (biggest ALB win)
• Offload large/static responses to CloudFront — removes bytes from the ALB
• Flatten rule chains — keep the hot path within the first 10 free rule evaluations
• Use NLB for raw TCP at high connection rates — 32× better connection economics
• Reuse TLS connections on NLB — avoid the 50-handshakes/sec TLS ceiling
• Right-size deregistration/idle timeouts — fewer half-open connections inflating the active-connection dimension
• Consolidate low-traffic ALBs — each ALB carries the ~$0.0225/hour base regardless of traffic

12. GCP Cloud Load Balancing

Global Anycast LB

GCP's external HTTP(S) LB is global — a single IP is announced from all Google PoPs worldwide via Anycast BGP. Traffic is terminated at the nearest Google edge, then forwarded over Google's private backbone to backends.

graph TD
    classDef user fill:#34495e,stroke:#212f3c,color:#fff
    classDef pop fill:#e67e22,stroke:#ba6018,color:#fff
    classDef backend fill:#2980b9,stroke:#1f618d,color:#fff

    subgraph ANYCAST["Single anycast IP announced from every Google PoP worldwide"]
        UserUS["User — US East"]:::user -->|"Anycast 34.x.x.x<br/>routed to nearest PoP"| PoP_NY["Google PoP<br/>New York"]:::pop
        UserEU["User — EU West"]:::user -->|"Anycast 34.x.x.x<br/>routed to nearest PoP"| PoP_AM["Google PoP<br/>Amsterdam"]:::pop
    end

    subgraph BACKBONE["Google's private backbone, not the public internet"]
        PoP_NY -->|"Terminated at edge,<br/>forwarded over backbone"| BS["Backend Service<br/>us-central1"]:::backend
        PoP_AM -->|"Terminated at edge,<br/>forwarded over backbone"| BS2["Backend Service<br/>europe-west1"]:::backend
    end

Components

URL Map Example

# host: api.example.com path: /v1/* → backend-service-v1
# host: api.example.com path: /v2/* → backend-service-v2
# default → backend-service-main

NEGs — Network Endpoint Groups

NEGs decouple the LB from instance groups. Instead of routing to a VM, the LB routes to an endpoint (IP:port).

Container-native LB (GKE + zonal NEG): traffic goes directly to pods, bypassing kube-proxy iptables. Lower latency, accurate health checks at pod level.

Premium vs Standard Tier

13. nginx as Load Balancer

Basic Upstream Config

upstream backend_pool {
    least_conn;                          # algorithm: least connections
    keepalive 32;                        # idle keepalive connections to upstream

    server 10.0.1.1:8080 weight=3;
    server 10.0.1.2:8080 weight=1;
    server 10.0.1.3:8080 backup;        # only used if others are down
}

server {
    listen 80;
    worker_processes auto;              # one worker per CPU core

    location / {
        proxy_pass         http://backend_pool;
        proxy_http_version 1.1;
        proxy_set_header   Connection "";     # enable keepalive to upstream
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_connect_timeout 5s;
        proxy_read_timeout    60s;
    }
}

Health Checks

nginx OSS has passive health checks only (marks backend down after failed proxy attempts):

upstream backend_pool {
    server 10.0.1.1:8080 max_fails=3 fail_timeout=30s;
}

nginx Plus (commercial) adds active health checks:

# nginx Plus only
upstream backend_pool {
    zone backend 64k;
    server 10.0.1.1:8080;
    health_check interval=10s fails=2 passes=3 uri=/healthz;
}

Community workaround: use nginx_upstream_check_module (Tengine patch) or route health check via a separate Lua block.

14. HAProxy

Config Structure

global
    maxconn     50000
    nbthread    4          # one thread per CPU

defaults
    mode        http
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    option      redispatch        # retry on another server if connection fails
    retries     3

frontend http_in
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/example.pem
    acl is_api  path_beg /api/
    acl is_beta hdr(X-Beta) -i true
    use_backend api_pool  if is_api
    use_backend beta_pool if is_beta
    default_backend web_pool

backend api_pool
    balance leastconn
    option  httpchk GET /healthz
    server  api1 10.0.1.1:8080 check inter 10s fall 2 rise 3
    server  api2 10.0.1.2:8080 check inter 10s fall 2 rise 3

backend web_pool
    balance roundrobin
    cookie  SERVERID insert indirect nocache
    server  web1 10.0.2.1:8080 check cookie web1
    server  web2 10.0.2.2:8080 check cookie web2

Stick Tables (Sticky Sessions without Cookies)

backend api_pool
    stick-table type ip size 100k expire 30m
    stick on src                  # route same src IP to same server

Stats Page

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:secret

ACLs

HAProxy ACLs are powerful pattern-matching on any request attribute:

acl is_mobile   hdr_sub(User-Agent) -i mobile
acl large_body  req.body_size gt 1048576
acl safe_method method GET HEAD OPTIONS

15. Comparison Table

16. Common Issues

502 Bad Gateway

LB received an invalid or empty response from backend.

Causes:

• Backend process crashed or not listening on the target port
• Backend returned a response the LB couldn't parse (protocol mismatch — e.g., backend sent HTTP/2 but LB expected HTTP/1.1)
• Backend closed the connection before sending a response (idle timeout race)
• Health check passes but app throws 502 for the specific path

Debug:

# Check backend is actually listening
ss -tlnp | grep :8080

# Check ALB access logs for target_status_code
aws s3 cp s3://my-alb-logs/... - | grep ' 502 '

# curl directly to backend IP bypassing LB
curl -v http://10.0.1.1:8080/api/healthz

504 Gateway Timeout

LB timed out waiting for the backend to respond.

Causes:

• Slow database query, external API call, or CPU-bound operation
• LB idle timeout < request processing time (ALB default: 60s)
• Backend threadpool exhausted, requests queued

Fix:

# Increase ALB idle timeout if request legitimately takes longer
aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn arn:... \
  --attributes Key=idle_timeout.timeout_seconds,Value=120

Thundering Herd on Deploy

When a new deployment starts, all LBs simultaneously shift traffic to new pods. If new pods are slow to warm up (JVM, model loading), they get overwhelmed.

Fix:

• Use readiness probes — pod not added to LB until /readyz returns 200
• Use minReadySeconds on Deployment — wait N seconds after pod Ready before counting it as available
• Use slow start (nginx: slow_start=30s per upstream server; HAProxy: slowstart)
• Canary deploy — send 5% traffic to new pods first

Connection Reset on Draining

Backend removed from pool while client had an established persistent connection → Connection reset by peer or ECONNRESET.

Fix:

• Set deregistration_delay long enough for in-flight requests to complete
• Backends should handle SIGTERM gracefully: stop accepting new connections, finish in-flight, then exit
• Set connection: close header in final responses during shutdown

// Go graceful shutdown
srv.Shutdown(context.WithTimeout(ctx, 30*time.Second))

Asymmetric Routing with NLB

NLB passes the real client IP (src IP preserved). If the backend's response route does not go back through the NLB (e.g., backend has a default route pointing to a different gateway), TCP session breaks because the client sees an unexpected source IP for the return packet.

Fix:

• Ensure backend instances route traffic destined to the client back through the NLB or same path
• Or enable source NAT on NLB (NLB target group: preserve_client_ip = false) — but you lose the real client IP
• Use security groups that allow the NLB's IP range, not just client IPs

Read Order

For networking context: osi-model.md → tcp-udp.md → tls-encryption.md → this file

For Kubernetes LB internals: ../kubernetes/kube-proxy-modes.md → ../kubernetes/networking.md

For AWS-specific: ../aws/request-flow-alb-to-pod.md → ../aws/README.md