MongoDB Internals

How MongoDB actually stores, replicates, and serves documents underneath mongosh — the storage engine, the oplog, how an election really resolves, and the write-concern arithmetic that decides whether a failover loses data. For the operational side of running this on real VMs (firewall rules, mongod.conf, the Prometheus exporter, a step-by-step bootstrap), see on-prem-vm/mongodb.md; for the Kubernetes-operator version, see on-prem-k8s/mongodb.md.

WiredTiger Storage Engine

Every write goes through the same three structures, whether it's a single insertOne or a bulk load:

graph TD
    classDef engine fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef mem fill:#3498db,stroke:#2471a3,color:#fff
    classDef durable fill:#e67e22,stroke:#ba6018,color:#fff
    classDef disk fill:#7f8c8d,stroke:#616a6b,color:#fff

    CLIENT["MongoDB client"] -->|"insert / update / delete"| MONGOD["mongod process"]
    MONGOD --> WT["WiredTiger Storage Engine"]:::engine

    subgraph WT["WiredTiger"]
        CACHE["WT Cache — in-memory B-tree pages<br/>size: wiredTigerCacheSizeGB<br/>default: 50% of (RAM − 1GB)"]:::mem
        JOURNAL["Journal (WAL)<br/>append-only, fsynced every 100ms<br/>or immediately if j:true"]:::durable
        CHECKPOINT["Checkpoint<br/>every 60s or 2GB of journal written<br/>flushes dirty cache pages to disk"]:::durable
    end

    CACHE -->|"dirty pages"| CHECKPOINT
    CHECKPOINT --> DATA["Collection files (.wt)<br/>B-tree, snappy-compressed by default"]:::disk
    CHECKPOINT --> IDX["Index files (.wt)<br/>B-tree on _id + user indexes"]:::disk
    JOURNAL -.->|"replayed on crash recovery<br/>if newer than last checkpoint"| DATA

The journal and the checkpoint solve two different failure windows. A crash between checkpoints only loses what's replayable from the journal — that's why the journal is fsynced far more often (100ms) than a full checkpoint runs (60s). Without the journal, a crash mid-checkpoint could lose everything written since the previous checkpoint, not just the last 100ms.

Sizing the cache

Document Model and BSON

MongoDB stores documents as BSON (Binary JSON), not JSON text:

BSON document: {_id: ObjectId("..."), name: "Alice", age: 30}

Binary encoding:
[doc_length 4B][type 1B][key "name\0"][value "Alice"][type 1B][key "age\0"][value 30 4B]...[terminator 0x00]

Why BSON, not JSON: the length-prefixed encoding means WiredTiger (and any driver) can skip over a field without parsing its contents — it just reads the 4-byte length and jumps. BSON also has native types JSON lacks: Date, ObjectId, Binary, Decimal128 — a JSON encoding would have to represent all of these as strings and lose type fidelity.

ObjectId is 12 bytes: 4-byte timestamp + 5-byte random value + 3-byte incrementing counter. That construction is deliberate — it's sortable by creation time, globally unique without a central sequence generator, and safe to generate on any node (including offline clients) without coordination.

Replication: Oplog, Elections & Quorum

The oplog

The oplog is a capped collection in the local database (local.oplog.rs). Every write on the primary is recorded as an idempotent operation — replaying the same oplog entry twice produces the same result, which is what makes replica catch-up and resync safe to retry.

sequenceDiagram
    participant P as Primary — local.oplog.rs
    participant S1 as Secondary 1
    participant S2 as Secondary 2

    Note over P: client write commits locally first
    P->>P: append idempotent entry to oplog
    par tailing cursor per secondary
        S1->>P: tail oplog (long-lived cursor, not polling)
        P-->>S1: stream new entries as they're appended
        S1->>S1: apply entry, advance local optime
    and
        S2->>P: tail oplog (long-lived cursor)
        P-->>S2: stream new entries
        S2->>S2: apply entry, advance local optime
    end
    Note over S1,S2: each secondary's "optime" is how far it has replayed —<br/>this is exactly what rs.status() reports as replication lag

Op types: i (insert), u (update), d (delete), c (command — createCollection, dropCollection), n (no-op / keepalive, also used to fill in a stable point for chained replication).

Oplog window: capped at a fixed size — default 5% of disk space, minimum ~1 GB. If a secondary falls behind by more than the oplog window (its next needed entry has already rolled off), it can no longer catch up incrementally and must perform a full initial resync instead.

Elections and quorum

A replica set elects a primary by majority vote among voting members. An arbiter is a voting member that holds no data — it exists purely to break ties cheaply, without paying for a full extra data copy.

graph TD
    classDef primary fill:#e74c3c,stroke:#c0392b,color:#fff
    classDef secondary fill:#3498db,stroke:#2471a3,color:#fff
    classDef arbiter fill:#f39c12,stroke:#ba6018,color:#fff

    HB["Secondary misses primary heartbeat<br/>(default electionTimeoutMillis: 10s)"] --> NOM["Eligible secondary calls for election<br/>(nominates itself as candidate)"]
    NOM --> VOTE["Every voting member casts one vote<br/>higher priority + more up-to-date optime wins ties"]
    VOTE --> MAJ{"Candidate got<br/>a majority of votes?"}
    MAJ -->|Yes| NEWP["Candidate becomes PRIMARY<br/>starts accepting writes"]:::primary
    MAJ -->|No — split vote or<br/>no majority reachable| RETRY["Election fails, cluster stays without<br/>a primary; retried after a randomized backoff"]

    subgraph "3-member set example"
        P["Primary (1 vote)"]:::primary
        S["Secondary (1 vote)"]:::secondary
        A["Arbiter (1 vote, no data)"]:::arbiter
    end

Why votes ≠ data copies: an arbiter's vote counts exactly the same as a data-bearing secondary's vote when computing whether a write reached w: "majority", or whether an election has a quorum — but it holds zero bytes of actual data. A 3-member Primary-Secondary-Arbiter (PSA) set only needs 2 of 3 votes to elect a new primary or acknowledge a majority write, and primary + arbiter alone satisfies that majority — even though the arbiter can never serve that data if the primary then dies.

Try It Yourself: Live PSA Election

This is a different election model than the Raft demo elsewhere in this repo (replication.md) — worth being precise about the difference rather than treating "leader election" as one interchangeable mechanic. Raft's demo picks whichever node happens to time out first and wins purely on term number and majority; the first candidate to campaign wins as long as its log qualifies. MongoDB's election is priority-weighted: each member has a configured priority (default 1, arbiters always 0), a member only calls an election after missing heartbeats for electionTimeoutMillis (default 10s), and a higher-priority secondary that is otherwise healthy can trigger its own election and take over from a lower-priority primary even with no failure at all — a "priority takeover," not just a race to time out first. The demo below models that, plus the PSA topology's specific risk: an arbiter votes but holds no data and can never itself become primary.

Default topology is PSA: P1 (priority 1, data), S1 (priority 1, data, starts as PRIMARY), A1 (arbiter — priority 0, no data). Click any node to kill or revive it, use "Kill Primary" to force a failover, or add a differently-prioritized secondary and watch it take over. Majority here is computed against the full configured voting membership (not just currently up nodes) — this is what makes the risk case below possible: kill the arbiter, then kill either remaining data node, and the cluster has only 1 of 3 votes left and cannot elect anyone.

Rollback on rejoin

If the old primary had writes that were never replicated to any secondary before it crashed, and a new primary was elected and continued accepting new writes, the two oplogs have diverged. When the old primary rejoins as a secondary, MongoDB rolls back its un-replicated writes — moving them out to a rollback directory as BSON files rather than silently discarding them — so it can resync onto the new primary's oplog history.

Write Concern and Read Concern — Deep Dive

Write concern controls how many replica set members must acknowledge a write before the client is told it succeeded. Getting this wrong is the single most common cause of "the failover ate my data."

sequenceDiagram
    participant APP as Application
    participant PRI as Primary
    participant SEC as Secondary

    Note over APP,SEC: writeConcern: {w: "majority", j: true}
    APP->>PRI: insert document
    PRI->>PRI: write to journal (j:true = fsync before ack)
    PRI->>SEC: replicate via oplog tailing
    SEC->>SEC: write to journal + apply
    SEC-->>PRI: acknowledge
    Note over PRI: majority (2/3 votes) confirmed
    PRI-->>APP: write result: confirmed

    Note over APP,SEC: readConcern: "majority"
    APP->>PRI: find({_id: ...}) with majority read concern
    PRI->>PRI: only return data already committed to a majority
    Note over PRI: won't return data that could still be rolled back
    PRI-->>APP: document (guaranteed stable, won't vanish on failover)

Aggregation Pipeline

An aggregation is a sequence of stages, each one transforming the document stream from the previous stage:

graph LR
    COLL["orders collection"] --> MATCH["$match<br/>{status:'paid', created_at:{$gte:...}}"]
    MATCH --> LOOKUP["$lookup<br/>join users by user_id"]
    LOOKUP --> UNWIND["$unwind<br/>flatten joined array"]
    UNWIND --> GROUP["$group<br/>{_id:'$user_id', total:{$sum:'$amount'}}"]
    GROUP --> MATCH2["$match<br/>post-group filter: total >= 1000"]
    MATCH2 --> SORT["$sort<br/>{total:-1}"]
    SORT --> LIMIT["$limit: 100"]
    LIMIT --> OUT["Result stream"]

db.orders.aggregate([
    {$match:  {status:"paid", created_at:{$gte: new Date("2024-01-01")}}},  // filter early, index-eligible
    {$lookup: {from:"users", localField:"user_id", foreignField:"_id", as:"user"}},
    {$unwind: "$user"},
    {$group:  {_id:"$user_id", total:{$sum:"$amount"}, count:{$sum:1}}},
    {$match:  {total:{$gte:1000}}},   // post-group filter, no index available here
    {$sort:   {total:-1}},
    {$limit:  100},
    {$project:{_id:0, user_id:"$_id", total:1, count:1}}
], {allowDiskUse: true})   // needed once an intermediate stage's working set exceeds 100MB RAM

explain('executionStats') on an aggregation shows exactly which stages used an index (IXSCAN) versus a full scan — the same distinction as a plain find().

Index Types

// Standard B-tree
db.users.createIndex({email: 1})                    // ascending
db.users.createIndex({email: 1, status: 1})         // compound
db.users.createIndex({email: 1}, {unique: true})    // unique

// Partial index — only indexes matching documents, smaller and faster
db.users.createIndex({email: 1}, {
    partialFilterExpression: {deleted: {$exists: false}}
})

// TTL index — auto-deletes documents N seconds after createdAt
db.sessions.createIndex({createdAt: 1}, {expireAfterSeconds: 3600})

// Text index — full-text search
db.posts.createIndex({body: "text", title: "text"})
db.posts.find({$text: {$search: "kubernetes failover"}})

// Geospatial
db.locations.createIndex({coords: "2dsphere"})
db.locations.find({coords: {$near: {$geometry: {type:"Point", coordinates:[77.2,28.6]}, $maxDistance: 1000}}})

explain() — Query Analysis

// Find execution plan
db.orders.find({user_id: "123", status: "paid"}).explain("executionStats")
// winningPlan.stage: "COLLSCAN" = full scan (bad) | "IXSCAN" = index (good)
// executionStats.totalDocsExamined vs totalDocsReturned: large ratio = missing index

// Compound index matching the query shape
db.orders.createIndex({user_id: 1, status: 1})

// Covering index: every projected field is in the index itself — zero document fetches
db.orders.createIndex({user_id: 1, status: 1, amount: 1})
db.orders.find({user_id: "123"}, {status: 1, amount: 1, _id: 0})
// explain shows "Using index" — no heap/document reads at all

Transactions (4.0+)

const session = db.getMongo().startSession();
session.startTransaction({ readConcern: {level: "snapshot"}, writeConcern: {w: "majority"} });
try {
    const accounts = session.getDatabase("bank").accounts;
    accounts.updateOne({_id: "alice"}, {$inc: {balance: -100}}, {session});
    accounts.updateOne({_id: "bob"},   {$inc: {balance:  100}}, {session});
    session.commitTransaction();
} catch (err) {
    session.abortTransaction();
    throw err;
} finally {
    session.endSession();
}

readConcern: "snapshot" gives every read inside the transaction a consistent point-in-time view — as if the whole transaction ran instantaneously — even though the two updateOne calls execute sequentially.

Change Streams

A real-time feed of insert/update/delete events, built directly on top of the oplog.

const stream = db.orders.watch([
    {$match: {"operationType": {$in: ["insert", "update"]}}},
    {$match: {"fullDocument.status": "paid"}}
]);
stream.on("change", change => processOrder(change.fullDocument));

// Crash recovery: persist the resumeToken, restart from exactly where you left off
const stream2 = db.orders.watch([], {resumeAfter: lastToken});

Sharding

graph TD
    classDef router fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef cfg fill:#f39c12,stroke:#ba6018,color:#fff
    classDef shard fill:#2980b9,stroke:#1f618d,color:#fff

    APP["Application"] --> MONGOS["mongos router<br/>stateless — no data of its own"]:::router
    MONGOS -->|"chunk map lookup"| CFG["Config server replica set<br/>authoritative chunk-to-shard mapping"]:::cfg
    MONGOS -->|"routes query to the<br/>right shard(s) only"| S1["Shard 1 replica set<br/>user_id: 0 – 500K"]:::shard
    MONGOS --> S2["Shard 2 replica set<br/>user_id: 500K – 1M"]:::shard
    CFG -.->|"balancer moves chunks<br/>to keep shards even"| S1
    CFG -.-> S2

sh.enableSharding("mydb")
sh.shardCollection("mydb.orders", {user_id: "hashed"})
// hashed  = even write distribution, but range queries scatter across every shard
// ranged  = supports efficient range queries, but monotonically increasing keys hotspot one shard

sh.getBalancerState()  // is the auto-balancer currently moving chunks?

Try It Yourself: Live Chunk Split & Migration

Pick a shard-key type, then insert values yourself. Try a monotonically increasing sequence (1, 2, 3, 4, …) under each mode and watch what happens to where the documents land — this is the same mechanism the toggle above describes and the quiz below tests, just running live instead of asserted in prose. Once one shard is visibly overloaded, click "Run balancer" and watch it split that shard's biggest chunk and migrate half of it to the least-loaded shard.

Monitoring

db.currentOp({secs_running: {$gt: 5}})   // find long-running ops
db.killOp(opid)
db.setProfilingLevel(1, {slowms: 100})   // log queries slower than 100ms
db.system.profile.find().sort({ts: -1}).limit(10)
rs.status()                              // replica set health + per-member replication lag