Hot Partitions

A distributed database is only as fast as its busiest partition. Skew turns a cluster of ten nodes into one node doing the work while nine watch.

The signature

Adding nodes does not help. That is the distinguishing symptom — a capacity problem improves with capacity, and skew does not.

Look for uneven per-node metrics while the data distribution looks even:

nodetool status                    # Cassandra: Load differs, Owns is equal

// MongoDB: chunk counts per shard.
db.getSiblingDB("config").chunks.aggregate([
  { $group: { _id: "$shard", chunks: { $sum: 1 } } }
]);

-- ClickHouse: rows per shard via the distributed table.
SELECT hostName(), count() FROM clusterAllReplicas(main, currentDatabase(), events_local)
GROUP BY hostName();

For Redis Cluster, compare operations per second per node; for ScyllaDB, compare per-shard reactor utilisation, since one saturated core is invisible in a node average.

Two different problems

Access skew — one partition receives disproportionate traffic. A celebrity user, a default tenant, a popular product.

Size skew — one partition holds disproportionate data. Unbounded growth for one key, which eventually makes reads and maintenance on that partition expensive regardless of traffic.

They have different fixes, and a partition can have both.

Fixes

Add a bucket to the key. Time is the usual choice, and it bounds growth as well as spreading load:

-- Before: unbounded, and hot for active devices.
PRIMARY KEY (device_id, ts)
 
-- After: one partition per device per day.
PRIMARY KEY ((device_id, day), ts)

Add a synthetic shard for a genuinely hot key. Write to key:{0..15} chosen at random, read all 16 and merge. Fan-out on read in exchange for spreading writes.

Hash a sequential key. A monotonically increasing shard key — timestamp, auto-increment id — always concentrates current writes on one partition. Hash it, or put a well-distributed column first.

Cache the hot key. For read-hot data, serving it from a cache or from replicas removes the load without touching the model.

Detecting it before production

Cluster metrics show skew after it exists. Instrument the application to record the partition key it accesses, sampled, so the distribution is visible in advance:

metrics.counter("db.access", tags={"partition": partition_key_bucket}).increment()

Then check the distribution during load testing, with production-shaped data — not with uniformly random test data, which hides exactly the skew you are looking for.