Apache Kafka on Kubernetes (Strimzi Operator)

Running Kafka as a StatefulSet on Kubernetes — broker identity, PVC binding, failover, and the operational sequences (pod rescheduling, rolling restarts, scaling) that a plain stateless Deployment never has to deal with.

Architecture

graph TD
    subgraph K8S["Kubernetes (GKE)"]
        STRIMZI["Strimzi Operator<br/>manages Kafka cluster lifecycle"]
        subgraph KAFKA["Kafka Cluster (StatefulSet)"]
            B0["kafka-0 (broker)<br/>leader for partitions: 0,3,6"]
            B1["kafka-1 (broker)<br/>leader for partitions: 1,4,7"]
            B2["kafka-2 (broker)<br/>leader for partitions: 2,5,8"]
        end
        subgraph ZK["ZooKeeper (or KRaft)"]
            Z0["zookeeper-0"]
            Z1["zookeeper-1"]
            Z2["zookeeper-2"]
        end
        STRIMZI --> KAFKA & ZK
        PVC_B0["PVC: data-kafka-0<br/>500Gi"]
        PVC_B1["PVC: data-kafka-1<br/>500Gi"]
        PVC_B2["PVC: data-kafka-2<br/>500Gi"]
        B0 --> PVC_B0
        B1 --> PVC_B1
        B2 --> PVC_B2
    end
    PROD["Producer"] --> B0 & B1 & B2
    B0 & B1 & B2 --> CONS["Consumer Groups"]

Broker Pod Rescheduling & PVC Reattachment

Kafka brokers run as a StatefulSet, not a Deployment — that distinction is the whole reason Kafka works on Kubernetes at all. A Deployment's pods are interchangeable; a StatefulSet's aren't. Each pod gets a stable, ordinal-indexed name (kafka-0, kafka-1, kafka-2) and a stable network identity via a headless Service, and each ordinal owns its own PVC from volumeClaimTemplates — data-kafka-1 belongs to kafka-1 for the life of the StatefulSet, not the life of any one pod.

That matters because it's what makes a rescheduled broker safe: the replacement pod keeps the same name, the same DNS entry, and reattaches to the same PVC — it comes back as the same broker with its own data still on disk, not a fresh empty node that has to be re-added to the cluster from scratch.

sequenceDiagram
    participant Node as K8s Node (kafka-1)
    participant Sched as StatefulSet Controller
    participant PVC as PVC: data-kafka-1
    participant B1 as kafka-1 (new pod)
    participant Cluster as Rest of cluster

    Note over Node: Node fails / pod evicted (OOM, drain, node loss)
    Node--xSched: kafka-1 pod terminated
    Sched->>Sched: notices desired replica count violated
    Sched->>B1: schedule replacement pod, same name "kafka-1"
    B1->>PVC: mounts existing PVC data-kafka-1 (not a new volume)
    Note over B1: local segments on disk are intact — no full re-sync needed
    B1->>Cluster: rejoins as follower via existing headless-service DNS name
    Cluster->>B1: leader sends only what kafka-1 missed while down
    B1->>Cluster: caught up, rejoins ISR

Two things have to be true for this to work in practice: the storage class must support the volume reattaching wherever the new pod lands (a ReadWriteOnce cloud block volume works; anything tied to one specific node would not survive a node loss), and terminationGracePeriodSeconds needs to be generous enough for Kafka to flush and close segments cleanly rather than being SIGKILLed mid-write.

Walk through the sequence step by step:

Topic Replication and ISR

graph TD
    TOPIC["Topic: orders<br/>partitions: 9<br/>replication.factor: 3"]

    P0["Partition 0<br/>Leader: kafka-0"] --> R0A["Replica: kafka-1 (ISR)"]
    P0 --> R0B["Replica: kafka-2 (ISR)"]

    P1["Partition 1<br/>Leader: kafka-1"] --> R1A["Replica: kafka-0 (ISR)"]
    P1 --> R1B["Replica: kafka-2 (ISR)"]

ISR (In-Sync Replicas): Replicas that are fully caught up with the leader (within replica.lag.time.max.ms = 10000ms). A replica falls out of ISR if it falls behind.

min.insync.replicas: Minimum ISR count required for a write to succeed. With replication.factor=3 and min.insync.replicas=2: a write succeeds if at least 2 replicas (including leader) acknowledge it. One broker can fail with zero data loss.

# Topic config
min.insync.replicas: 2
replication.factor: 3

# Producer config for guaranteed delivery
acks: all             # wait for all ISR replicas to confirm
retries: 2147483647   # retry indefinitely
enable.idempotence: true  # exactly-once semantics

Partition Leader Election (Failover)

sequenceDiagram
    participant ZK2 as ZooKeeper
    participant B0_2 as kafka-0 (was leader)
    participant B1_2 as kafka-1 (ISR replica)
    participant CONT as Strimzi Operator

    Note over B0_2: kafka-0 crashes (node failure or OOM)
    B0_2--xZK2: ZooKeeper session expires (6s default)
    ZK2->>ZK2: partition 0 has no leader
    ZK2->>B1_2: Elect kafka-1 as partition 0 leader
    B1_2->>B1_2: Becomes leader for partition 0
    Note over B1_2: Producers/consumers reconnect to new leader

    Note over CONT: kafka-0 recovers
    CONT->>B0_2: kafka-0 starts as follower
    B0_2->>B1_2: Fetch from leader, catch up
    B0_2->>ZK2: Joins ISR after catching up

Step through the same failover as discrete stages:

Unclean leader election: If ALL ISR replicas are down, Kafka can optionally elect an out-of-sync replica (unclean.leader.election.enable=true). Never enable in production — guarantees data loss.

Strimzi Kafka Cluster YAML

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka
spec:
  kafka:
    version: 3.6.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      log.retention.hours: 168       # 7 days retention
      log.segment.bytes: 1073741824  # 1GB segments
    storage:
      type: persistent-claim
      size: 500Gi
      class: premium-rwo             # GKE SSD storage class
    resources:
      requests:
        memory: 8Gi
        cpu: 2
      limits:
        memory: 16Gi
        cpu: 4
    jvmOptions:
      -Xms: 4096m
      -Xmx: 4096m

  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 50Gi
      class: premium-rwo

Rolling Restarts

Any change to the Kafka custom resource that requires a broker restart — a version bump, a JVM option, a value in the spec.kafka.config block — triggers a rolling restart, not a full-cluster bounce. Strimzi restarts one broker pod at a time, in ordinal order, and waits for that broker to rejoin the ISR for every partition it holds before touching the next one.

That one-at-a-time discipline is what keeps min.insync.replicas satisfied throughout the whole operation: with replication.factor=3 and min.insync.replicas=2, taking down one broker at a time still leaves 2 in the ISR for every partition, so acks=all producers keep working uninterrupted. Restarting two brokers concurrently could drop a partition's ISR below min.insync.replicas and start failing writes mid-rollout — exactly why Strimzi enforces the sequential order instead of restarting pods in parallel.

Scaling the Broker StatefulSet

Changing spec.kafka.replicas scales the StatefulSet, but "add a broker" and "remove a broker" aren't mirror images of each other — one is close to free, the other requires manual data movement first.

Consumer Groups and Lag

graph LR
    TOPIC2["Topic: orders<br/>9 partitions"] --> CG1["Consumer Group: payments<br/>3 consumers<br/>3 partitions each"]
    TOPIC2 --> CG2["Consumer Group: analytics<br/>1 consumer<br/>9 partitions"]

# Check consumer group lag
kubectl exec -it kafka-0 -- kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group payments

# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          1500            1500            0
# orders  1          1480            1520            40   <- lagging!
# orders  2          1600            1600            0

# Alert if lag > threshold
# Prometheus: kafka_consumer_group_lag > 1000

Backups

Kafka doesn't have a built-in backup mechanism. Options:

1. MirrorMaker 2 — replicate topics to another cluster (GKE → GCS via Kafka Connect)
2. Kafka Connect S3 Sink — stream all messages to GCS/S3
3. Volume snapshots — snapshot PVCs (consistent only if broker is stopped first)

# Kafka Connect S3 Sink (backup all topics to GCS)
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
spec:
  class: io.confluent.connect.s3.S3SinkConnector
  config:
    topics: ".*"
    s3.bucket.name: my-kafka-backup
    s3.region: us-central1
    flush.size: 1000
    rotate.interval.ms: 3600000  # rotate files every hour