MongoDB on Kubernetes
0/0 checks
Replica Set Architecture
graph TD
subgraph "MongoDB Replica Set (StatefulSet: 3 nodes)"
P["mongo-0 PRIMARY<br/>accepts all reads+writes<br/>holds oplog"]
S1["mongo-1 SECONDARY<br/>replicates from primary<br/>can serve reads (readPreference)"]
S2["mongo-2 SECONDARY (or Arbiter)<br/>vote-only if arbiter<br/>or full copy"]
end
P -->|"oplog streaming<br/>(async by default)"| S1
P -->|"oplog streaming"| S2
CLIENT["App"] -->|"PRIMARY connection"| P
READER["Read-only app"] -->|"readPreference: secondary"| S1
Oplog (Operations Log): MongoDB's replication log. A capped collection in the local database on the primary. Secondaries tail the oplog and apply operations. Size determines how far behind a secondary can fall before it needs a full resync.
mongo-2 is configured as an arbiter instead of a full secondary. It has a vote in elections — does it also hold a copy of the data, and can an app read from it?
No. An arbiter is vote-only — it never holds a copy of the data or the oplog. It exists purely to break ties in elections (useful for keeping an odd number of voters without paying for a third full data-bearing node). Since it has no data, it can never be a read target, no matter what readPreference is set.
Write Concern and Read Concern
# Write concern: how many nodes must confirm a write
writeConcern:
w: "majority" # majority of voting members must acknowledge
j: true # writes must be journaled (flushed to disk)
wtimeout: 5000 # fail if not confirmed in 5s
# Read concern: what data reads can see
readConcern:
level: "majority" # only read data committed to majority (no dirty reads)
# Alternatives: local (default, may read uncommitted), linearizable
Write concern majority with 3 nodes: 2/3 nodes must confirm. If primary + 1 secondary confirm → write committed. If primary fails after 1 secondary confirms → the secondary has the data and becomes primary.
Default. Returns whatever data is on the node handling the read, even if that data hasn't been replicated to (or could still be rolled back by) the rest of the set. Fastest, but a dirty read is possible.
Only returns data that's been acknowledged by a majority of voting members — the same durability bar as write concern
majority. Can't return data that could later be rolled back after a failover.
Strongest guarantee: the read reflects every write that completed before it started, cluster-wide. Only meaningful on reads against the primary, and the slowest of the three because it has to confirm no concurrent election is in progress.
A write with writeConcern w: "majority" is acknowledged by the primary and exactly one secondary, then the primary immediately crashes. Is that write lost?
No. Primary + 1 secondary = 2 of 3 nodes, which already satisfies majority before the crash. The secondary that has the data is a valid candidate to become the new primary in the election that follows, so the write survives the failover. It would only be at risk if fewer than a majority had confirmed it.
Election and Failover
sequenceDiagram
participant P2 as mongo-0 (Primary)
participant S1_2 as mongo-1 (Secondary)
participant S2_2 as mongo-2 (Secondary)
Note over P2: Primary becomes unavailable (node failure)
S1_2->>S2_2: heartbeat missed for electionTimeoutMillis (10s)
S1_2->>S1_2: Increment term, become candidate
S1_2->>S2_2: RequestVote (term=2)
S2_2-->>S1_2: VoteGranted (I'm up to date)
Note over S1_2: Won majority (2/3 votes including self)
S1_2->>S1_2: Become PRIMARY
Note over S1_2,S2_2: Election complete, ~10s downtime
Note over P2: mongo-0 recovers
P2->>S1_2: I'm alive, current term?
S1_2-->>P2: Term=2, I am primary
P2->>P2: Step down, become secondary
P2->>S1_2: Catch up oplog
Walk through the same failover as discrete stages:
1. Stable. mongo-0 is primary, accepting all reads and writes. mongo-1 and mongo-2 are secondaries, tailing its oplog.
2. Primary goes dark. mongo-0 fails (node loss, crash). The secondaries stop receiving heartbeats from it.
3. Election timeout, candidacy. After
electionTimeoutMillis (10s default) with no heartbeat, mongo-1 increments the term, becomes a candidate, and requests votes from the rest of the set.
4. Majority vote, new primary. mongo-2 grants its vote since mongo-1's oplog is at least as recent as its own. mongo-1 wins a majority (2 of 3, including itself) and becomes the new primary — roughly 10s of write unavailability for that period.
5. Old primary rejoins as secondary. mongo-0 recovers, discovers a higher term already has a primary, steps down to secondary, and catches up its oplog from mongo-1.
Election prerequisites: Candidate must have oplog at least as recent as the majority. A secondary that is too far behind cannot win election — prevents data loss.
mongo-2's oplog has fallen noticeably behind the rest of the set. The primary fails and mongo-2 tries to call an election and become primary. What stops it?
A candidate must have an oplog at least as recent as the majority of the set to win an election. Because mongo-2 is behind, the other voting members won't grant it their votes, so it can't win — this is what prevents a stale node from becoming primary and silently losing whatever writes it never received.
Ops Manager / Community Operator
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongodb
spec:
members: 3
type: ReplicaSet
version: "7.0.0"
security:
authentication:
modes: ["SCRAM"]
users:
- name: appuser
db: admin
passwordSecretRef:
name: mongodb-secret
roles:
- name: readWrite
db: myapp
statefulSet:
spec:
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: premium-rwo
resources:
requests:
storage: 100Gi
Backups
# Consistent backup with mongodump (logical)
mongodump \
--uri="mongodb://user:pass@mongo-0:27017,mongo-1:27017,mongo-2:27017/?replicaSet=rs0" \
--readPreference=secondary \ # don't impact primary
--oplog \ # capture oplog for point-in-time
--out /backup/$(date +%Y%m%d)
# Restore from dump
mongorestore --uri="mongodb://..." --oplogReplay /backup/20240115
# VolumeSnapshot (physical — faster, consistent)
# Must pause writes or use --fsync lock for consistency
kubectl exec mongo-0 -- mongosh --eval "db.fsyncLock()"
# Take snapshot...
kubectl exec mongo-0 -- mongosh --eval "db.fsyncUnlock()"
Reads documents out through the driver and writes them back out as BSON — portable across MongoDB versions and storage engines, and safe to run against a live secondary with
--readPreference=secondary so it doesn't add load to the primary. Slower than a disk-level copy, and restoring means replaying documents back in, not just remounting a volume.
A raw copy of the data files — much faster to take and restore for large datasets, but only consistent if writes are quiesced first (
fsyncLock) or the node is stopped. Tied to the same storage engine and, generally, the same MongoDB version it was taken from.
Why does the VolumeSnapshot approach require db.fsyncLock() before the snapshot, when mongodump doesn't need anything like it?
mongodump reads consistent, already-committed documents out through the normal driver interface — there's nothing "mid-write" to catch. A VolumeSnapshot copies the raw data files on disk, so without fsyncLock pausing writes first, the snapshot could capture files mid-write and be internally inconsistent on restore. fsyncUnlock releases the write pause once the snapshot is taken.
Read Preference Options
| Mode | Where reads go | Use case |
|---|---|---|
primary (default) |
Always primary | Strong consistency required |
primaryPreferred |
Primary if available, else secondary | Slight performance boost with fallback |
secondary |
Any secondary | Analytics, reporting (stale OK) |
secondaryPreferred |
Secondary if available, else primary | Read scale-out |
nearest |
Lowest latency node | Geographic distribution |
Every secondary in the set is down (only the primary is reachable). A query uses readPreference: secondary. Another uses secondaryPreferred. What happens to each?
secondary has nowhere to fall back to — with no secondary available, the read fails outright. secondaryPreferred falls back to the primary when no secondary is available, so it still succeeds. The "Preferred" suffix is exactly what makes the fallback behavior different, not just a synonym for the base mode.
MongoDB on Kubernetes — Community Operator
# Install MongoDB Community Operator
helm repo add mongodb https://mongodb.github.io/helm-charts
helm install community-operator mongodb/community-operator \
--namespace mongodb-operator --create-namespace
# Secret for admin password
apiVersion: v1
kind: Secret
metadata:
name: mongodb-secret
namespace: mongodb
type: Opaque
stringData:
password: "changeme"
---
# 3-node Replica Set
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: mongodb
namespace: mongodb
spec:
members: 3
type: ReplicaSet
version: "7.0.4"
security:
authentication:
modes: ["SCRAM"]
users:
- name: appuser
db: admin
passwordSecretRef:
name: mongodb-secret
roles:
- name: readWrite
db: myapp
- name: clusterMonitor
db: admin
scramCredentialsSecretName: appuser-scram
# MongoDB configuration
additionalMongodConfig:
operationProfiling:
slowOpThresholdMs: 100
replication:
oplogSizeMB: 2048
# Storage per pod
statefulSet:
spec:
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes: [ReadWriteOnce]
storageClassName: gp3
resources:
requests:
storage: 100Gi
- metadata:
name: logs-volume
spec:
accessModes: [ReadWriteOnce]
storageClassName: gp3
resources:
requests:
storage: 10Gi
template:
spec:
containers:
- name: mongod
resources:
requests:
cpu: "2"
memory: 8Gi
limits:
cpu: "4"
memory: 16Gi
# Check replica set status
kubectl get mongodbcommunity mongodb -n mongodb
# NAME PHASE VERSION
# mongodb Running 7.0.4
# Connect string (headless service creates per-pod DNS)
# mongodb-0.mongodb-svc.mongodb.svc.cluster.local:27017
# mongodb-1.mongodb-svc.mongodb.svc.cluster.local:27017
# mongodb-2.mongodb-svc.mongodb.svc.cluster.local:27017
kubectl exec -it mongodb-0 -n mongodb -- mongosh \
"mongodb://appuser:changeme@mongodb-0.mongodb-svc:27017,mongodb-1.mongodb-svc:27017,mongodb-2.mongodb-svc:27017/myapp?replicaSet=mongodb"
# Check replica set status from inside
> rs.status()
> rs.isMaster() # shows who is primary
# Services created automatically by operator:
# mongodb-svc ClusterIP None — headless, per-pod DNS
# mongodb-svc-ext ClusterIP — single endpoint for the replica set
apiVersion: v1
kind: Service
metadata:
name: mongodb-svc
namespace: mongodb
spec:
clusterIP: None # headless
selector:
app: mongodb-svc
ports:
- port: 27017
The operator creates both mongodb-svc (headless, per-pod DNS) and mongodb-svc-ext (a single ClusterIP endpoint). Why does the connection string above list out mongodb-0.mongodb-svc, mongodb-1.mongodb-svc, mongodb-2.mongodb-svc individually instead of just pointing at the single mongodb-svc-ext endpoint?
A replica set client needs to know about — and be able to reach — every member individually, because it has to track who's currently primary and be able to route reads to specific secondaries under a given readPreference. A single virtual endpoint can't distinguish one member from another, so the driver is handed the per-pod headless DNS names and figures out the topology itself via replicaSet=mongodb.