Databases on Kubernetes (GKE / On-Prem)

Running stateful databases on Kubernetes — system design, replication, failover, snapshots, and operational patterns for each database.

0/0 checks

Why Run DBs on K8s?

Running databases on Kubernetes is operationally harder than managed services (Cloud SQL, RDS) but gives you:

Cost control — no managed service markup (3-5× cheaper at scale)
Portability — same setup on GKE, EKS, on-prem
Customization — specific versions, plugins, tuning impossible in managed offerings
Data residency — compliance requirements that forbid managed cloud DBs

When NOT to run DBs on K8s: Small teams, < 3 engineers who understand K8s storage, or when managed services fit your compliance requirements. Operational burden is real.

A 2-engineer team wants Postgres on K8s to avoid the managed-service markup. Good idea?

Files

File	Database	Topics
postgres.md	PostgreSQL	StatefulSet, Patroni HA, sync/async replication, automatic failover, PITR, pgbackup
mysql.md	MySQL	Operator, InnoDB Cluster, semi-sync replication, master promotion, XtraBackup
mongodb.md	MongoDB	Replica set on K8s, elections, oplog, readPreference, mongodump/mongorestore
redis-cluster.md	Redis Cluster	Sharding, slot distribution, sentinel vs cluster, persistence (RDB/AOF), failover
kafka.md	Apache Kafka	Strimzi operator, partition replication, ISR, leader election, consumer groups
clickhouse.md	ClickHouse	ClickHouse Operator, sharding, replication via ZooKeeper/ClickHouse Keeper, backups

Common Patterns Across All DBs

graph TD
    PRIMARY["Primary / Leader<br/>(accepts writes)"] -->|"sync or async replication"| REP1["Replica 1<br/>(read traffic)"]
    PRIMARY --> REP2["Replica 2<br/>(standby / DR)"]

    PROBE["Health check / probe<br/>detect primary failure"] -->|"primary unhealthy"| ELECT["Leader election<br/>replica promotion"]
    ELECT --> NEW_PRIMARY["New Primary<br/>(former Replica 1)"]
    NEW_PRIMARY -->|"old primary rejoins as replica"| OLD["Old Primary<br/>(now replica)"]

Sync vs Async Replication

Write completes when: the primary AND at least one replica confirm the write.

Data loss on failover: zero — the replica already has everything the primary had.

Write latency: higher — every write waits on a round trip to the replica.

Replica lag: zero, by construction.

Use case: financial data, and anything else where losing a committed write is not an option.

Write completes when: the primary confirms; replicas catch up afterward.

Data loss on failover: up to the replication lag — seconds to minutes of writes can vanish.

Write latency: lower — the primary never waits on a replica.

Replica lag: can fall behind under load or network pressure.

Use case: read replicas, analytics, DR copies where a little staleness is acceptable.

Your replica is 40 seconds behind the primary under async replication, and the primary just died. What happens to the last 40 seconds of writes?

Snapshot Strategy (3-2-1 Rule)

graph LR
    LIVE["Live Database<br/>(primary)"] -->|"daily full backup"| S3["Object Storage<br/>(GCS / S3)"]
    LIVE -->|"WAL/oplog streaming"| S3
    S3 -->|"cross-region copy"| S3_DR["DR Region<br/>Object Storage"]
    LIVE -->|"volume snapshot"| SNAP["K8s VolumeSnapshot<br/>(GCP Persistent Disk)"]

1. Full snapshot. Daily full backup of the primary, retained for 7 days.

2. Continuous incremental. WAL (Postgres) or oplog (MongoDB) streamed continuously, enabling point-in-time recovery (PITR) between full snapshots.

3. Cross-region copy. At least one copy of every backup lives in a different region or zone than the primary — the "1 offsite" in 3-2-1.

4. Test restore. A weekly automated restore, run end-to-end. An untested backup is not a backup.

You have a daily full snapshot and continuous WAL streaming, but you've never actually restored from either. Are you covered?