Caching — Deep Dive Reference

From fundamentals to production patterns.

1. Why Cache

Read amplification: a single app query can fan out to dozens of DB reads. A cache short-circuits this.

Cost reduction: DB CPU is expensive; Redis/Memcached nodes are cheap per RPS served.

2. Cache Tier Hierarchy

Each tier trades latency for sharing scope: the faster a cache is, the fewer things can see what's in it. CPU cache is invisible outside one core; an in-process cache is invisible outside one app instance; only the distributed tier is actually shared state across a fleet.

graph TD
    classDef client fill:#7f8c8d,stroke:#616a6b,color:#fff
    classDef edge fill:#8e44ad,stroke:#6c3483,color:#fff
    classDef app fill:#3498db,stroke:#2471a3,color:#fff
    classDef cache fill:#e67e22,stroke:#ba6018,color:#fff
    classDef db fill:#2c3e50,stroke:#1a252f,color:#fff
    classDef cpu fill:#27ae60,stroke:#1e8449,color:#fff

    Client["Client Browser<br/>issues the request"]:::client

    subgraph EDGE["Edge Tier — global, per-PoP"]
        CDN["CDN Edge Cache<br/>~5ms, geographic"]:::edge
    end

    subgraph APP["Application Tier — per instance"]
        CPU["CPU L1/L2/L3 Cache<br/>1–40ns, per core"]:::cpu
        AppL1["App In-Process Cache<br/>~0.01ms, per instance heap"]:::app
    end

    subgraph SHARED["Shared Cache Tier — fleet-wide"]
        Redis["Distributed Cache<br/>Redis / Memcached<br/>~0.5ms, shared"]:::cache
    end

    subgraph DBT["Database Tier"]
        DBQueryCache["DB Query Cache<br/>~1ms, per DB node"]:::db
        DB["Database Disk<br/>~10ms+"]:::db
    end

    Client -->|"request"| CDN
    CDN -->|"miss: forward"| AppL1
    CPU -.->|"used transparently<br/>by the app process"| AppL1
    AppL1 -->|"miss: fan out"| Redis
    Redis -->|"miss: query"| DBQueryCache
    DBQueryCache -->|"miss: disk read"| DB

3. Cache-Aside (Lazy Loading)

The application owns the cache interaction.

sequenceDiagram
    participant App
    participant Cache
    participant DB

    App->>Cache: GET key
    alt cache hit
        Cache-->>App: return value
        Note over App: DB never touched — fast path
    else cache miss (cold key or expired TTL)
        Cache-->>App: nil
        App->>DB: SELECT ...
        DB-->>App: row data
        App->>Cache: SET key value EX ttl
        Note over Cache: key now warm until TTL expiry
        App-->>App: return value
    end
    Note over App,DB: Risk: if another process updates the DB directly,<br/>this cache entry goes stale until TTL expiry

Code pattern (Python/Redis):

def get_user(user_id):
    key = f"user:{user_id}"
    data = redis.get(key)
    if data:
        return json.loads(data)
    user = db.query("SELECT * FROM users WHERE id = %s", user_id)
    redis.setex(key, 300, json.dumps(user))
    return user

Pros: Only caches what's actually read. Cache failures don't break writes.

Cons: First read is always slow (cold miss). Stale data possible if DB changes outside the app.

Thundering herd risk: Many requests miss simultaneously on cold start or expiry — all hit DB at once. See §11.

4. Write-Through

Every write goes to cache AND DB in the same operation.

sequenceDiagram
    participant App
    participant Cache
    participant DB
    participant App2 as Another App Instance

    App->>Cache: SET key value
    activate Cache
    Cache->>DB: INSERT/UPDATE (synchronous)
    DB-->>Cache: ack
    Cache-->>App: ack
    deactivate Cache
    Note over App,DB: Write latency = cache write + DB write,<br/>paid on every write regardless of future reads

    App2->>Cache: GET key
    Cache-->>App2: return value (already fresh)
    Note over App2: No stale read possible —<br/>cache was updated in the same operation as the DB

Pros: Cache is always consistent with DB. No stale reads after writes.

Cons: Write latency = cache latency + DB latency. Cache fills with data that may never be read.

5. Write-Behind (Write-Back)

Write to cache immediately; flush to DB asynchronously.

sequenceDiagram
    participant App
    participant Cache
    participant Queue as Dirty-Key Queue
    participant DB

    App->>Cache: SET key value
    Cache-->>App: ack (fast — DB not involved yet)
    Cache->>Queue: enqueue dirty key
    Note over Queue: keys batch up until<br/>flush interval or batch size hit

    par periodic flush
        Queue->>DB: batched INSERT/UPDATE
        DB-->>Queue: ack
        Queue->>Queue: mark keys clean
    end

    Note over Cache,DB: Risk: if Cache/Queue crashes before flush,<br/>queued writes are lost — the client already got "ack"

Pros: Extremely fast writes. Batch DB writes reduce I/O.

Cons: Data loss if cache crashes before flush. Complexity in failure handling.

6. Read-Through

Cache sits in front of DB and fetches transparently on miss.

sequenceDiagram
    participant App
    participant Cache as Cache + Loader<br/>(e.g. Spring @Cacheable)
    participant DB

    App->>Cache: GET key
    alt cache hit
        Cache-->>App: value
    else cache miss
        Note over Cache,DB: The cache itself calls the DB —<br/>App code never sees this branch
        Cache->>DB: SELECT ...
        DB-->>Cache: data
        Cache->>Cache: store fetched value
        Cache-->>App: data
    end
    Note over App: App's code is identical on hit or miss —<br/>always just "GET key"

Difference from cache-aside: App only talks to cache. Cache library/provider handles DB fetching. Example: Spring Cache with @Cacheable.

7. All 4 Patterns — Side-by-Side

Solid arrows are synchronous (the caller waits); dashed arrows are asynchronous (the caller already moved on). That one visual distinction is most of what separates write-through's safety from write-behind's speed.

graph LR
    classDef app fill:#3498db,stroke:#2471a3,color:#fff
    classDef cache fill:#e67e22,stroke:#ba6018,color:#fff
    classDef db fill:#2c3e50,stroke:#1a252f,color:#fff

    subgraph CacheAside["Cache-Aside — app owns both paths"]
        CA_App["App"]:::app -->|"read: GET"| CA_Cache["Cache"]:::cache
        CA_App -->|"on miss: SELECT"| CA_DB["DB"]:::db
        CA_App -->|"on miss: SET"| CA_Cache
        CA_App -->|"write: UPDATE"| CA_DB
    end

    subgraph WriteThrough["Write-Through — sync, consistent"]
        WT_App["App"]:::app -->|"write: SET"| WT_Cache["Cache"]:::cache
        WT_Cache -->|"sync write"| WT_DB["DB"]:::db
    end

    subgraph WriteBehind["Write-Behind — fast, eventually consistent"]
        WB_App["App"]:::app -->|"write: SET"| WB_Cache["Cache"]:::cache
        WB_Cache -.->|"async flush"| WB_DB["DB"]:::db
    end

    subgraph ReadThrough["Read-Through — cache owns the DB fetch"]
        RT_App["App"]:::app -->|"read: GET"| RT_Cache["Cache"]:::cache
        RT_Cache -->|"on miss: SELECT"| RT_DB["DB"]:::db
    end

Live Demo: Cache vs DB, Same Two Operations, Three Strategies

Pick a strategy below, then Read/Write a key and watch what actually happens to the cache and the DB — they diverge only on the write path. Reads behave identically everywhere: a hit returns from cache, a miss falls through to the DB and repopulates the cache.

8. Eviction Policies

Algorithms

Redis maxmemory-policy options

# redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru

Rule of thumb: Use allkeys-lru for pure caches. Use volatile-ttl when mixing persistent and ephemeral keys.

9. Redis Data Structures for Caching

String — simple key-value

SET user:1001 '{"name":"alice","role":"admin"}' EX 300
GET user:1001
INCR page:views:home   # atomic counter

Hash — object fields without JSON serialization

HSET user:1001 name alice role admin age 30
HGET user:1001 name
HGETALL user:1001
# Update one field without fetching/re-serializing whole object
HSET user:1001 age 31

Sorted Set — leaderboards, rate limiting

# Leaderboard
ZADD leaderboard 9500 player:alice
ZADD leaderboard 8200 player:bob
ZREVRANGE leaderboard 0 9 WITHSCORES   # top 10

# Sliding window rate limit (requests in last 60s)
ZADD rate:user:1001 1719651233 req:uuid1
ZREMRANGEBYSCORE rate:user:1001 -inf (now-60)
ZCARD rate:user:1001   # current count

List — queues, recent activity

LPUSH recent:user:1001 "viewed:product:42"
LTRIM recent:user:1001 0 99   # keep last 100
LRANGE recent:user:1001 0 -1

Bitmap — feature flags, daily active users

# User 1001 active today (bit at offset = user_id)
SETBIT dau:2026-06-29 1001 1
BITCOUNT dau:2026-06-29   # total DAU
GETBIT dau:2026-06-29 1001

HyperLogLog — unique count (approximate, 0.81% error)

PFADD unique:visitors:home user:1001 user:1002 user:1001
PFCOUNT unique:visitors:home   # returns 2, not 3

Uses ~12KB regardless of cardinality — far cheaper than a Set for billions of IDs.

10. Redis vs Memcached

Choose Memcached when: pure string caching, extreme memory efficiency matters, multi-threaded performance on many cores.

Choose Redis when: you need persistence, replication, complex data structures, pub/sub, or Lua atomicity.

11. Cache Stampede / Thundering Herd

Problem: A hot key expires. Thousands of requests all miss simultaneously, all hit DB at once, DB collapses.

The danger isn't the miss itself — cache-aside is designed to tolerate misses. It's that a hot key means hundreds or thousands of in-flight requests were relying on that one cached value at the same moment, so the instant it disappears, all of them fall through to the DB together, as if the cache had never been there at all.

sequenceDiagram
    participant R1 as Request 1
    participant R2 as Request 2
    participant RN as Request N (thousands more)
    participant Cache
    participant DB

    Note over Cache: hot_key's TTL just expired — cache now empty for this key
    par near-simultaneous misses
        R1->>Cache: GET hot_key
        Cache-->>R1: nil (miss)
    and
        R2->>Cache: GET hot_key
        Cache-->>R2: nil (miss)
    and
        RN->>Cache: GET hot_key
        Cache-->>RN: nil (miss)
    end
    Note over R1,RN: all of them saw the same empty key within milliseconds —<br/>none knows another is about to run the identical query
    rect rgb(90, 30, 30)
    par stampede — every miss independently falls through to the DB
        R1->>DB: SELECT (recompute hot_key)
        R2->>DB: SELECT (recompute hot_key)
        RN->>DB: SELECT (recompute hot_key)
    end
    end
    Note over DB: DB was sized for a trickle of cache misses,<br/>not the full read volume at once — CPU/connections saturate

Solution 1: Mutex / Distributed Lock

def get_with_lock(key):
    val = redis.get(key)
    if val:
        return val
    lock_key = f"lock:{key}"
    if redis.set(lock_key, "1", nx=True, ex=5):  # acquire lock
        try:
            val = db.fetch(key)
            redis.setex(key, 300, val)
            return val
        finally:
            redis.delete(lock_key)
    else:
        time.sleep(0.05)        # wait and retry
        return get_with_lock(key)

Solution 2: XFetch (Probabilistic Early Expiration)

Recompute before expiry based on probability that increases as TTL decreases.

import math, random, time

def xfetch(key, ttl, beta=1.0):
    value, delta, expiry = cache_get_with_metadata(key)
    if time.time() - beta * delta * math.log(random.random()) >= expiry:
        # probabilistically recompute early
        value = db.fetch(key)
        cache_set_with_metadata(key, value, ttl)
    return value

Solution 3: Stale-While-Revalidate

Return stale value immediately; refresh in background.

def get_stale_ok(key, ttl, stale_ttl=60):
    val = redis.get(key)
    if val:
        return val
    # Check extended stale window
    stale = redis.get(f"stale:{key}")
    if stale:
        threading.Thread(target=refresh, args=(key, ttl)).start()
        return stale
    return refresh_sync(key, ttl)

Solution 4: Background Refresh

Cron/worker refreshes hot keys before they expire. Works well for known-hot keys.

12. Cache Warming

Why it matters: A cold cache after deploy → all traffic hits DB → DB overload → cascading failure.

Strategies

Pre-warming on deploy (best for known hot keys):

# During deploy pipeline, before shifting traffic
python warm_cache.py --keys top_products,homepage,config

Lazy warming: Cache-aside naturally warms on first reads. Acceptable if DB can handle initial cold traffic.

Scheduled warming job:

# K8s CronJob, runs every 5 min for known-hot queries
def warm():
    top_products = db.query("SELECT * FROM products ORDER BY views DESC LIMIT 100")
    for p in top_products:
        redis.setex(f"product:{p.id}", 600, json.dumps(p))

Shadow traffic replay: Replay recent production logs against new cache before cutover.

Rule: Never deploy a stateful cache service without a warm-up phase. Use feature flags or canary to shift traffic gradually.

13. Cache Invalidation Patterns

"There are only two hard things in computer science: cache invalidation and naming things." — Phil Karlton

TTL-Based

Simplest. Set EX on every key. Accept eventual staleness.

SET product:42 '...' EX 300

Event-Driven Invalidation

DB triggers or application events delete cache entries on writes.

def update_product(product_id, data):
    db.execute("UPDATE products SET ... WHERE id = %s", product_id)
    redis.delete(f"product:{product_id}")          # invalidate
    redis.delete(f"product:list:category:{data.category_id}")  # invalidate list

Tag-Based Invalidation

Group related keys under a tag; invalidate all by tag.

# On write: record tag → keys mapping
redis.sadd("tag:category:5", "product:42", "product:43")
# On category update:
keys = redis.smembers("tag:category:5")
redis.delete(*keys)
redis.delete("tag:category:5")

Version Keys (Cache Busting)

Append a version to the key. Old keys expire naturally via TTL.

version = redis.get("product:42:version") or 1
key = f"product:42:v{version}"
data = redis.get(key)
# To invalidate:
redis.incr("product:42:version")

14. Distributed Cache Consistency

The Write Race Problem

Two app instances update the same key simultaneously — last write wins but may overwrite a newer value.

sequenceDiagram
    participant A as App Instance A
    participant B as App Instance B
    participant Cache

    rect rgb(90, 30, 30)
    Note over A,B: Unsafe — plain GET + SET, no coordination
    A->>Cache: GET user:1 → v1
    B->>Cache: GET user:1 → v1
    A->>Cache: SET user:1 v2
    B->>Cache: SET user:1 v3 (B never saw v2, overwrites blindly)
    Note over Cache: v2 is gone — B's write didn't know v2 existed
    end

    rect rgb(30, 70, 40)
    Note over A,B: Safe — WATCH / MULTI / EXEC (optimistic locking)
    A->>Cache: WATCH user:1, GET user:1 → v1
    B->>Cache: WATCH user:1, GET user:1 → v1
    A->>Cache: MULTI, SET user:1 v2, EXEC
    Cache-->>A: EXEC succeeds — user:1 unchanged since A's WATCH
    B->>Cache: MULTI, SET user:1 v3, EXEC
    Cache-->>B: EXEC returns nil — user:1 changed since B's WATCH
    B->>Cache: retry: GET user:1 → v2, recompute, MULTI/EXEC
    Note over Cache: no write silently lost — B is forced to retry on fresh data
    end

Optimistic locking doesn't prevent the race from happening — B still reads the stale v1 first. What it prevents is B's write from landing unnoticed: EXEC fails atomically the moment the watched key changed underneath it, forcing B back through the read-transform-write loop until it succeeds against current data.

Redis WATCH / MULTI / EXEC (Optimistic Locking)

WATCH user:1
val = GET user:1
MULTI
  SET user:1 <new_value>
EXEC   # returns nil if user:1 changed since WATCH → retry

def safe_update(key, transform):
    with redis.pipeline() as pipe:
        while True:
            try:
                pipe.watch(key)
                current = pipe.get(key)
                new_val = transform(current)
                pipe.multi()
                pipe.set(key, new_val)
                pipe.execute()
                break
            except redis.WatchError:
                continue  # retry on conflict

Compare-and-Swap with Lua (atomic)

-- SET only if value matches expected
local current = redis.call('GET', KEYS[1])
if current == ARGV[1] then
  redis.call('SET', KEYS[1], ARGV[2])
  return 1
end
return 0

EVAL "<script>" 1 user:1 expected_val new_val

15. CDN vs Application Cache vs DB Cache

16. Sizing & Monitoring

Working Set Estimation

working_set = hot_keys × avg_value_size × replication_factor
# Example: 1M hot keys × 2KB avg × 2 replicas = 4 GB

Hit Rate Math

hit_rate = keyspace_hits / (keyspace_hits + keyspace_misses)
effective_latency = hit_rate × cache_latency + (1 - hit_rate) × db_latency

# Example: 90% hit, 1ms cache, 20ms DB:
# = 0.9 × 1 + 0.1 × 20 = 0.9 + 2 = 2.9ms avg
# vs no cache: 20ms avg

Redis Monitoring

redis-cli INFO stats | grep -E "keyspace_hits|keyspace_misses|evicted_keys|expired_keys"
redis-cli INFO memory | grep used_memory_human
redis-cli INFO keyspace

Key metrics:

• keyspace_hits / keyspace_misses → compute hit rate; target > 90%
• evicted_keys > 0 → memory pressure, increase maxmemory or evict more aggressively
• used_memory_rss >> used_memory → memory fragmentation (see §17)
• connected_clients near maxclients (default 10000) → connection pool exhaustion

Prometheus / Grafana

Use redis_exporter. Key PromQL:

# Hit rate
rate(redis_keyspace_hits_total[5m]) /
  (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# Eviction rate
rate(redis_evicted_keys_total[5m])

# Memory fragmentation ratio (>1.5 = fragmented)
redis_memory_used_rss_bytes / redis_memory_used_bytes

17. Common Issues

Cache Poisoning

Attacker inserts malicious data into cache (e.g., via HTTP response splitting, unvalidated key construction).

Fix: Validate and sanitize all keys. Never build cache keys from raw user input without normalization. Use signed tokens for sensitive cache values.

Stale Reads After Write

Write updates DB but forgets to invalidate cache. Reads return old data.

Fix: Always pair DB writes with cache invalidation in the same transaction scope (or use write-through). Use short TTLs as a safety net.

Memory Fragmentation

Redis allocates memory in chunks; after many DEL/EXPIRE cycles, RSS grows above used_memory.

# Check fragmentation ratio (>1.5 is concerning)
redis-cli INFO memory | grep mem_fragmentation_ratio

# Enable active defragmentation (Redis 4+)
# redis.conf
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10   # start at 10% fragmentation
active-defrag-threshold-upper 100

Hot Keys

A single key receives millions of requests/sec, saturating a single Redis shard.

Detection:

redis-cli --hotkeys   # requires maxmemory-policy != noeviction
redis-cli monitor | head -100   # live command stream

Fix: Local L1 shadow cache in each app instance for the hot key (small TTL, 1–5s):

local_cache = {}  # in-process dict

def get_hot(key):
    if key in local_cache and time.time() < local_cache[key]['exp']:
        return local_cache[key]['val']
    val = redis.get(key)
    local_cache[key] = {'val': val, 'exp': time.time() + 1}  # 1s local TTL
    return val

Alternative: replicate hot keys across multiple Redis nodes with a prefix shard (hot:0:key, hot:1:key, ...) and randomly select a shard on read.

Connection Pool Exhaustion

Each app thread holding a Redis connection; pool depleted under load.

Fix:

# redis-py with connection pool
pool = redis.ConnectionPool(host='redis', port=6379, max_connections=50)
r = redis.Redis(connection_pool=pool)

• Set max_connections to (app_threads × 0.5) as a starting point
• Monitor connected_clients in Redis
• Use pipelining to batch commands and reduce round-trips

Quick Reference

Cache-aside    → app manages read/write, lazy population
Write-through  → sync write to cache + DB, no stale reads
Write-behind   → fast writes, async DB flush, data loss risk
Read-through   → cache handles DB fetch transparently

LRU            → general purpose
LFU            → skewed/Zipf access patterns
volatile-ttl   → mixed persistent + ephemeral keys

Stampede fix   → mutex lock OR XFetch probabilistic OR stale-while-revalidate
Invalidation   → TTL (simple) → event-driven (consistent) → tag-based (grouped)
Hot key fix    → L1 local shadow cache with short TTL
Fragmentation  → activedefrag yes in redis.conf