Redis Internals

How Redis actually stores data in memory, decides what to evict, persists it to disk, and keeps replicas and cluster nodes in sync underneath the command you type into redis-cli — the per-type internal encodings, the fork-based persistence model, and the failure-mode arithmetic that decides whether a cluster keeps serving traffic through a node or an AZ loss.

Data Structures Under the Hood

Redis is not just a key-value store. Each data type has a specific internal encoding that changes based on size for memory efficiency.

graph TD
    classDef compact fill:#27ae60,stroke:#1e8449,color:#fff
    classDef general fill:#2980b9,stroke:#1f618d,color:#fff
    classDef stream fill:#8e44ad,stroke:#6c3483,color:#fff

    subgraph STR["String"]
        SINT["int<br/>value fits in a long"]:::compact
        SEMB["embstr<br/>&lt;=44 bytes, one allocation"]:::compact
        SRAW["raw<br/>&gt;44 bytes, separate allocation"]:::general
    end

    subgraph LST["List"]
        LLP["listpack<br/><=128 elements, <=64 bytes each"]:::compact
        LQL["quicklist<br/>linked list of listpacks"]:::general
        LLP -->|"exceeds list-max-listpack-size"| LQL
    end

    subgraph HSH["Hash"]
        HLP["listpack<br/><=128 fields, <=64 bytes each"]:::compact
        HHT["hashtable<br/>beyond threshold"]:::general
        HLP -->|"exceeds hash-max-listpack-entries"| HHT
    end

    subgraph ST["Set"]
        SIS["intset<br/>pure integers, sorted array"]:::compact
        SLP["listpack<br/><=128 small mixed elements"]:::compact
        SHT["hashtable<br/>mixed types or large"]:::general
        SIS -->|"non-integer element added"| SLP
        SLP -->|"exceeds set-max-listpack-entries"| SHT
    end

    subgraph ZS["Sorted Set (ZSet)"]
        ZLP["listpack<br/><=128 elements, <=64 bytes each"]:::compact
        ZSK["skiplist + hashtable<br/>beyond threshold"]:::general
        ZLP -->|"exceeds zset-max-listpack-entries"| ZSK
    end

    subgraph STM["Stream"]
        STRD["Radix tree of listpacks<br/>append-only log with consumer groups"]:::stream
    end

Why listpack first? Dense memory layout — all elements contiguous. Cache-friendly. Converts to hashtable/skiplist when it exceeds thresholds (configurable via hash-max-listpack-entries).

Skiplist for sorted sets: O(log n) insert/delete/rank. Alternative to B-tree for in-memory sorted data. Each node has random forward pointers at multiple levels.

Incremental rehashing: growing the hashtable without a latency spike

Once a hash (or Redis's own top-level keyspace dict, which is the same dict structure under the hood) is in hashtable mode, it keeps growing as more entries land in it. Growing a hashtable normally means allocating a bigger array and rehashing every existing key into it — for a dict with millions of keys, doing that in one blocking pass would stall every client for however long the full rehash takes, an O(n) latency spike landing on whichever unlucky command happened to trigger it.

Redis avoids that by rehashing incrementally. When the load factor crosses its threshold, it allocates a second table double the size of the first and keeps both tables alive at once instead of rehashing everything up front. From that point on, every read or write that touches the dict does a small amount of extra work first: it migrates exactly one bucket from the old table into the new one, then proceeds with whatever operation was actually requested. A cursor tracks which old-table bucket is migrated next, so the work marches forward one bucket per operation until the old table is fully drained, at which point it's freed and the new table becomes the only table. If the server is otherwise idle, a periodic cron job (serverCron) also nudges the migration along so a quiet keyspace still finishes a rehash in the background rather than waiting indefinitely for the next command. While a rehash is in progress, lookups have to check both tables — new writes go straight to the new table, but a key inserted before the rehash started might still be sitting in either one, so a GET can't skip the old table until migration is finished.

This is the mechanism that keeps Redis's per-command complexity at amortized O(1) even while a hashtable is actively resizing — the cost of moving the whole table is spread thinly across every subsequent operation instead of paid all at once, so no single command ever pays for more than one bucket's worth of migration.

Try it below: the load factor here is total keys divided by the old table's bucket count, and it's set to trigger a rehash the moment it exceeds 1.0, doubling the bucket count each time. Insert enough keys and Search mid-rehash to see both tables checked and the migration cursor advance one bucket per operation.

Memory Model

graph TD
    classDef io fill:#16a085,stroke:#117a65,color:#fff
    classDef core fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef mem fill:#3498db,stroke:#2471a3,color:#fff
    classDef data fill:#7f8c8d,stroke:#616a6b,color:#fff

    CLIENTS["Connected clients"] --> IOT["I/O threads<br/>(Redis 6.0+, optional)<br/>read/parse/write sockets in parallel"]:::io
    IOT --> LOOP["Single-threaded event loop<br/>executes one command at a time"]:::core
    LOOP -.->|"atomic execution —<br/>no two commands interleave"| ATOMIC["Atomicity guarantee"]:::core
    LOOP --> MEM["Memory allocator<br/>jemalloc (default)"]:::mem
    MEM --> DICT["Main dictionary<br/>hash table of all keys<br/>+ separate expires dict for TTL keys"]:::data
    DICT --> OBJS["Redis objects<br/>encoded per data type (see above)"]:::data

Redis is single-threaded for command execution. I/O is handled by an event loop (like Node.js). Commands execute atomically — no two commands run concurrently.

Since Redis 6.0: I/O threads for reading/writing network (multi-threaded I/O), but command execution still single-threaded.

Persistence: RDB vs AOF

Redis offers two independent persistence mechanisms — a point-in-time snapshot (RDB) and a replayable write log (AOF) — and they solve different halves of the durability problem: RDB gives a fast-to-load, compact on-disk image; AOF gives a much smaller recovery window at the cost of a slower reload.

graph LR
    classDef rdb fill:#e67e22,stroke:#ba6018,color:#fff
    classDef aof fill:#2980b9,stroke:#1f618d,color:#fff
    classDef both fill:#27ae60,stroke:#1e8449,color:#fff

    WRITE["Write command executes"] --> AOFBUF
    WRITE -.->|"BGSAVE / save-interval rule<br/>/ replica full resync"| FORK

    subgraph RDBFLOW["RDB — point-in-time snapshot"]
        FORK["fork() child process<br/>copy-on-write of parent's memory"]:::rdb
        SNAP["Child serializes entire dataset<br/>to disk, unaffected by new writes"]:::rdb
        DUMP["dump.rdb<br/>compact binary format"]:::rdb
        FORK --> SNAP --> DUMP
    end

    subgraph AOFFLOW["AOF — append-only command log"]
        AOFBUF["Command appended to AOF buffer"]:::aof
        FSYNC["fsync policy: always / everysec / no"]:::aof
        AOFFILE["appendonly.aof<br/>human-readable command log"]:::aof
        REWRITE["BGREWRITEAOF<br/>fork() + rewrite compact equivalent"]:::aof
        AOFBUF --> FSYNC --> AOFFILE
        AOFFILE -.->|"grows unbounded over time"| REWRITE
    end

    subgraph RESTART["Restart — RDB + AOF together"]
        LOADRDB["Load dump.rdb<br/>(fast — binary, sequential read)"]:::both
        REPLAY["Replay AOF tail written<br/>since that snapshot"]:::both
        LOADRDB --> REPLAY
    end

    DUMP -.->|"loaded first on restart"| LOADRDB
    AOFFILE -.->|"tail replayed after RDB load"| REPLAY

RDB: fork + save, step by step

AOF rewrite, step by step

Eviction Policies

When maxmemory is reached, Redis uses an eviction policy:

CONFIG SET maxmemory 4gb
CONFIG SET maxmemory-policy allkeys-lru

Replication

The replication backlog is a bounded, in-memory ring buffer (repl-backlog-size) that the master appends every write to, independently of any specific replica. It's what makes a cheap partial resync possible at all — without it, every reconnect would have no choice but a full RDB transfer.

sequenceDiagram
    participant MASTER as Redis Master
    participant BACKLOG as Replication Backlog<br/>(bounded ring buffer)
    participant REPLICA as Replica

    Note over MASTER,REPLICA: Initial sync — replica has never connected before
    REPLICA->>MASTER: PSYNC ? -1
    MASTER->>MASTER: BGSAVE — fork, serialize dataset (copy-on-write)
    MASTER-->>REPLICA: +FULLRESYNC replication_id offset
    MASTER-->>REPLICA: RDB file (full snapshot)
    Note over REPLICA: Load RDB into memory
    par while RDB transfers
        MASTER->>MASTER: buffer new writes for this replica
        MASTER->>BACKLOG: also append every write here
    end
    MASTER-->>REPLICA: buffered commands accumulated during transfer
    Note over REPLICA: Apply buffered commands — now caught up

    Note over MASTER,REPLICA: Steady state — ongoing replication
    MASTER->>BACKLOG: append every write (bounded — oldest entries roll off)
    MASTER->>REPLICA: stream command (same format as AOF)
    REPLICA->>REPLICA: apply command, advance replication offset

    Note over MASTER,REPLICA: Replica disconnects, then reconnects
    REPLICA->>MASTER: PSYNC replication_id last_offset
    alt offset still inside the backlog
        MASTER-->>REPLICA: +CONTINUE — only the missed commands
        Note over REPLICA: Partial resync — cheap, no RDB transfer
    else offset already rolled off the backlog
        MASTER-->>REPLICA: +FULLRESYNC — starts over from the top
        Note over REPLICA: Falls back to the full BGSAVE + RDB transfer flow above
    end

Initial full resync, step by step

Replication is asynchronous. Replica may be behind. WAIT numreplicas timeout blocks until N replicas confirm offset — simulate synchronous replication.

Redis Cluster Internals

graph TD
    classDef master fill:#2980b9,stroke:#1f618d,color:#fff
    classDef compute fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef gossip fill:#8e44ad,stroke:#6c3483,color:#fff

    subgraph SLOTS["Hash slot routing"]
        KEY["SET order:123 data"]:::compute --> HASH2["CRC16('order:123') % 16384 = 7832"]:::compute
        HASH2 -->|"7832 falls in this range"| M2S["Master 2<br/>owns slots 5461-10922"]:::master
        M1S["Master 1<br/>owns slots 0-5460"]:::master
        M3S["Master 3<br/>owns slots 10923-16383"]:::master
        M1S -.- M2S -.- M3S
    end

    subgraph GOSSIP["Gossip protocol — every node learns every other node's state"]
        G1["Master 1"]:::gossip -->|"heartbeat + state<br/>every 100ms"| G2["Master 2"]:::gossip
        G2 -->|"heartbeat + state"| G3["Master 3"]:::gossip
        G3 -->|"heartbeat + state"| G1
    end

Failure detection, as a state machine:

stateDiagram-v2
    [*] --> OK
    OK --> PFAIL: heartbeat missed for cluster-node-timeout
    PFAIL --> OK: node responds again
    PFAIL --> FAIL: majority of masters also report PFAIL for this node
    FAIL --> OK: node rejoins and is reachable again

A single node marking another PFAIL is just one opinion — it takes a majority of masters independently reaching the same conclusion before the cluster treats it as an agreed, cluster-wide FAIL and starts a failover.

ASK vs MOVED redirects:

Lua Scripting (Atomic Operations)

# INCR with conditional limit — atomic via Lua
EVAL "
local current = redis.call('GET', KEYS[1])
if current and tonumber(current) >= tonumber(ARGV[1]) then
    return 0
end
return redis.call('INCR', KEYS[1])
" 1 rate:user:123 100

# Lua scripts execute atomically — no race conditions between GET and INCR

Key Metrics

# Memory usage (alert > 80% of maxmemory)
redis_memory_used_bytes / redis_memory_max_bytes > 0.8

# Hit rate (alert < 90%)
rate(redis_keyspace_hits_total[5m]) /
(rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) < 0.9

# Replica count (counts replicas, NOT lag — for lag use master_repl_offset − slave_repl_offset)
redis_connected_slaves < 1

# Evicted keys per second (should be 0 for non-cache workloads)
rate(redis_evicted_keys_total[1m]) > 0

# Command latency
redis_commands_duration_seconds_total / redis_commands_processed_total > 0.001

Pub/Sub

# Publisher
PUBLISH notifications '{"type":"order_paid","id":"123"}'

# Subscriber (blocks waiting for messages)
SUBSCRIBE notifications
# Or pattern-subscribe
PSUBSCRIBE order.*    # matches order.paid, order.cancelled, etc.

Pub/Sub is fire-and-forget — messages are lost if no subscriber is connected. For durability use Streams instead.

Redis Streams (Persistent Message Queue)

# Producer: append to stream
XADD orders * user_id 123 amount 99.99 status paid
# Returns: "1700000000000-0" (auto-generated ID: timestamp-sequence)

# Consumer group: multiple consumers, each gets different messages
XGROUP CREATE orders payments $ MKSTREAM

# Consumer reads (and acknowledges)
XREADGROUP GROUP payments consumer-1 COUNT 10 BLOCK 0 STREAMS orders >
# > means: give me undelivered messages for this consumer
XACK orders payments 1700000000000-0  # mark as processed

# Re-deliver messages not acknowledged after 30 seconds
XAUTOCLAIM orders payments consumer-1 30000 0-0

# Check pending (unacknowledged) messages
XPENDING orders payments - + 10

Streams = persistent, ordered, consumer groups, exactly-once delivery. Much more powerful than Pub/Sub.

Consumer group message lifecycle

Sentinel vs Cluster

graph TD
    classDef master fill:#2980b9,stroke:#1f618d,color:#fff
    classDef replica fill:#5dade2,stroke:#2e86c1,color:#fff
    classDef sentinel fill:#f39c12,stroke:#ba6018,color:#fff

    subgraph SENT["Sentinel — HA without sharding"]
        SM["Master<br/>all writes + reads"]:::master --> SR1["Replica 1"]:::replica
        SM --> SR2["Replica 2"]:::replica
        SENT1["Sentinel 1"]:::sentinel & SENT2["Sentinel 2"]:::sentinel & SENT3["Sentinel 3"]:::sentinel -->|"monitor + vote on failover"| SM
    end

    subgraph CLUS["Cluster — sharding + HA"]
        CM1["Master 1<br/>slots 0-5460"]:::master --> CR1["Replica 1"]:::replica
        CM2["Master 2<br/>slots 5461-10922"]:::master --> CR2["Replica 2"]:::replica
        CM3["Master 3<br/>slots 10923-16383"]:::master --> CR3["Replica 3"]:::replica
        CM1 -.->|"gossip"| CM2 -.->|"gossip"| CM3 -.->|"gossip"| CM1
    end

WAIT Command — Synchronous Replication

# Ensure at least N replicas have received writes before returning
SET key value
WAIT 1 1000  # wait for 1 replica, timeout 1000ms
# Returns: number of replicas that ACKed

# Simulate synchronous replication for critical writes
SET account:alice:balance 500
WAIT 1 500  # 0-RTT usually, blocks if replica is lagging

Key Expiry Internals

Redis uses two mechanisms to expire keys:

1. Lazy expiry: checks TTL only when the key is accessed — no CPU cost, but expired keys linger in memory
2. Active expiry: background job samples 20 random keys every 100ms, deletes expired ones, repeats if >25% were expired

Consequence: a key's TTL can expire but the key still occupies memory until accessed or the background job finds it. For memory-sensitive workloads, use maxmemory-policy to force eviction.

Try It Yourself: Lazy vs Active Expiry

Set a key with a short TTL (try 3 seconds), then leave it alone — watch its row turn red once the deadline passes ("expired, not yet swept"), and watch it disappear a moment later when the active-expiry cycle's next sweep catches it, with nobody ever calling Get. Or set one and hit "Get" on it right after the deadline to see the lazy path delete it on access instead. Either way the key eventually goes; which mechanism gets there first is basically a race against the 1-second sweep clock.

Pipeline and MULTI/EXEC

# Pipelining: send multiple commands without waiting for responses
# (network optimization — reduces round trips)
redis-cli --pipe << 'EOF'
SET key1 val1
SET key2 val2
INCR counter
EOF

# Transactions: atomic execution of multiple commands
MULTI
  SET account:alice 500
  SET account:bob 300
  INCR tx_counter
EXEC
# All three execute atomically — no other client's commands interleaved
# Unlike DB transactions: no rollback on command error (EXEC always runs all)

Debugging and Profiling

# Real-time command monitor (like tcpdump for Redis)
redis-cli MONITOR
# Shows every command as it arrives — use briefly, high overhead

# Slow log (commands > slowlog-log-slower-than microseconds)
redis-cli CONFIG SET slowlog-log-slower-than 10000  # 10ms
redis-cli SLOWLOG GET 10
# 1) 1) (integer) 14          # log entry ID
#    2) (integer) 1700000000  # timestamp
#    3) (integer) 15000       # execution time (microseconds)
#    4) 1) "KEYS"             # command + args (KEYS is O(n) — never use in prod)
#       2) "*"

# Memory analysis
redis-cli MEMORY USAGE mykey         # bytes used by one key
redis-cli MEMORY DOCTOR               # automated analysis
redis-cli --bigkeys                   # scan for large keys (run off-peak)
redis-cli --memkeys                   # sample keys by memory usage

Cluster Failure Modes

Slot coverage loss — the cluster goes read-only

Redis Cluster requires all 16384 hash slots to be covered by a reachable master. If a master goes down AND its replica fails to be promoted (or there is no replica), those slots become unavailable. The cluster refuses writes to uncovered slots.

# Check cluster health
redis-cli -h redis-node-1 -p 6379 CLUSTER INFO
# cluster_state:ok        ← healthy
# cluster_state:fail      ← one or more slots uncovered

# Find which slots are uncovered
redis-cli -h redis-node-1 -p 6379 CLUSTER NODES | grep fail
# <node-id> <ip>:6379 master,fail - ...   ← failed master

# Manual failover (if replica is running but hasn't promoted)
redis-cli -h <replica-host> -p 6379 CLUSTER FAILOVER
# or force (ignores replication lag — may lose recent writes)
redis-cli -h <replica-host> -p 6379 CLUSTER FAILOVER FORCE

Automatic failover, step by step

Recovery checklist:

# After failed node comes back
redis-cli -h redis-node-1 CLUSTER MEET <recovered-node-ip> 6379
redis-cli -h redis-node-1 CLUSTER REPLICATE <new-master-id>
# Resync: node downloads full RDB from master (can take minutes for large datasets)
redis-cli -h <recovered-node> REPLICATION  # watch master_sync_in_progress

Split-brain — cluster partitioned

Redis Cluster prevents split-brain by requiring quorum (majority of masters) to elect new masters. With 6 nodes (3 masters + 3 replicas), losing one AZ means:

• 1 master unreachable → its replica promotes ✓
• 2 masters unreachable (minority) → quorum lost → cluster goes down ✗

graph TD
    classDef alive fill:#27ae60,stroke:#1e8449,color:#fff
    classDef dead fill:#c0392b,stroke:#922b21,color:#fff
    classDef result fill:#7f8c8d,stroke:#616a6b,color:#fff

    START["3-master cluster"] --> LOSS["2 of 3 masters<br/>become unreachable simultaneously"]
    LOSS --> M1["Master 1"]:::dead
    LOSS --> M2["Master 2"]:::dead
    LOSS --> M3["Master 3<br/>(still alive)"]:::alive
    M3 --> QUORUM{"1 of 3 masters reachable —<br/>quorum needs 2 of 3"}
    QUORUM -->|"quorum NOT met"| RESULT["cluster_state:fail<br/>writes rejected cluster-wide"]:::result

Multi-AZ layout for resilience:

graph TD
    classDef master fill:#2980b9,stroke:#1f618d,color:#fff
    classDef replica fill:#5dade2,stroke:#2e86c1,color:#fff

    subgraph AZA["AZ-a"]
        M1["master-1<br/>slots 0-5460"]:::master
        R4["replica-4<br/>(replicates master-2)"]:::replica
    end
    subgraph AZB["AZ-b"]
        M2["master-2<br/>slots 5461-10922"]:::master
        R5["replica-5<br/>(replicates master-3)"]:::replica
    end
    subgraph AZC["AZ-c"]
        M3["master-3<br/>slots 10923-16383"]:::master
        R6["replica-6<br/>(replicates master-1)"]:::replica
    end

    M1 -.->|"replicated to"| R6
    M2 -.->|"replicated to"| R4
    M3 -.->|"replicated to"| R5

Replicas sit in a different AZ from their own master. An AZ loss then only ever takes one master and one other master's replica — never a master together with its own replica — so a promotion is always available and the surviving masters still clear quorum.

Hot key problem

A hot key is a single key receiving a disproportionate share of traffic. In Redis Cluster, a hot key maps to one slot → one master → that node becomes the bottleneck.

Detection:

# redis-cli hot key analysis (requires maxmemory-policy != noeviction)
redis-cli -h redis-node-1 --hotkeys
# OUTPUT: hot key 'user:session:12345' - freq: 50000/sec

# Monitor in real time
redis-cli -h redis-node-1 MONITOR | grep "GET\|SET" | head -100
# Warning: MONITOR is O(n) per command, use sparingly in production

# Use redis-cell or keydb for rate info
redis-cli -h redis-node-1 OBJECT FREQ <keyname>   # LFU policy only

Solutions:

1. Client-side caching (Redis 6+ tracking mode)
   Client caches value locally; server sends invalidation when key changes
   → reduces hot key traffic by 90%+ for mostly-read keys

2. Key sharding — append suffix to spread across slots
   "user:session" → "user:session:{0}", "user:session:{1}", ..., "user:session:{N}"
   Client randomly picks a shard; reads from any, writes to all
   → N shards = N nodes share the load
   N = 10-50 for very hot keys

3. Local in-process cache (L1)
   Store hot key in application memory (sync with Redis TTL)
   → near-zero latency, no network, but stale by TTL window

4. Read replicas
   READONLY command on replica allows reads
   → distributes read traffic across replica + master
   redis-cli -h <replica> READONLY
   GET user:session:12345   # served by replica

# Python: client-side sharding for hot key
import hashlib, random

def hot_key_get(redis_client, base_key: str, shards: int = 10) -> str:
    shard = random.randint(0, shards - 1)
    return redis_client.get(f"{base_key}:{shard}")

def hot_key_set(redis_client, base_key: str, value: str, shards: int = 10):
    pipe = redis_client.pipeline()
    for i in range(shards):
        pipe.set(f"{base_key}:{i}", value, ex=300)
    pipe.execute()

Sentinel vs Cluster — decision guide

Hash tags for multi-key ops in cluster:

Without hash tag:
  MGET user:1:name user:1:email  → may be on different slots → CROSSSLOT error

With hash tag (curly braces define the slot key):
  MGET {user:1}:name {user:1}:email  → both hash to "user:1" → same slot → works