Caching Pitfalls

A cache absorbs load until it stops absorbing it. These three patterns describe how that happens.

Cache Stampede

Also called the thundering herd. A popular key expires, and every concurrent request misses at once and goes to the origin database — which was sized for the cached request rate, not the raw one.

Locking. Only one request recomputes; the rest wait briefly and re-read.

def get_with_lock(key, ttl, compute):
    value = r.get(key)
    if value is not None:
        return value
 
    token = secrets.token_hex(16)
    if r.set(f"lock:{key}", token, nx=True, px=5_000):
        try:
            value = compute()
            r.set(key, value, ex=ttl)
            return value
        finally:
            r.eval(RELEASE_SCRIPT, 1, f"lock:{key}", token)
 
    time.sleep(0.05)                       # brief wait, then re-read
    return r.get(key) or compute()         # fall back rather than block forever

Note the fallback: a request must never wait indefinitely for another request's recomputation. See Distributed Locks for what this lock does and does not guarantee.

Probabilistic early expiry. Recompute slightly before expiry, with a probability that rises as the TTL approaches, so recomputations spread out naturally instead of synchronising.

Stale-while-revalidate. Store the value with a longer TTL than its logical freshness. When a request finds it logically stale, serve it immediately and refresh in the background. The origin never sees a stampede because there is always something to serve.

Cache Penetration

Requests for keys that do not exist in the cache or in the database. Every one reaches the origin, and nothing is ever cached, so the cache provides no protection at all. It is the shape of a scraping or enumeration attack.

Cache the negative result, with a short TTL:

value = r.get(key)
if value == NULL_SENTINEL:
    return None
if value is None:
    row = db.fetch(key)
    if row is None:
        r.set(key, NULL_SENTINEL, ex=60)   # short: the row may be created soon
        return None
    r.set(key, serialize(row), ex=3600)
    return row

Keep the negative TTL short so a legitimately created record becomes visible quickly.

Filter obviously invalid keys before touching either layer, and use a Bloom filter for the membership test when the key space is large and the miss rate is high.

Cache Avalanche

A large set of keys expires simultaneously — because they were created together with the same TTL, or because the cache was restarted or flushed. The origin receives the full uncached load at once.

• Add jitter to every TTL. ttl = base + random(0, base * 0.1) is enough to desynchronise.
• Warm the cache after a restart for the keys you know are hot, before opening traffic.
• Rate-limit or queue origin lookups so a cold cache degrades latency rather than taking the database down.

Design principle