Sharding

Sharding splits one logical dataset across several nodes, each holding a disjoint subset. It is the only way to scale writes beyond one machine, and it is a one-way door: choosing badly is expensive to correct.

Sharding is not partitioning

Partitioning splits a table into pieces within one node — mainly for query pruning and cheap data expiry. Sharding splits data across nodes — for capacity. Some engines use one word for both, which causes confusion in design discussions; be explicit about which one you mean.

Choosing a shard key

The shard key determines everything that follows. A good one satisfies four properties:

1. High cardinality. More distinct values than you will ever have shards.
2. Even distribution. No value carries a disproportionate share of traffic.
3. Present in the common queries. A query without the shard key hits every shard.
4. Stable. Changing a document's shard key means deleting and reinserting it in most engines.

These conflict. user_id distributes well and is present in user-scoped queries, but reporting queries that filter by date now fan out. A composite key of (tenant_id, created_at) supports tenant-scoped time ranges but concentrates load if one tenant dominates.

Hash versus range

Hash sharding is the safer default when queries are point lookups by the key. Range sharding is right when range scans dominate and you can accept managing skew.

What becomes harder

Everything that used to be a single-node operation now spans nodes:

• Joins. Cross-shard joins either move data or are unsupported. Denormalise, or co-locate related data by sharding both collections on the same key.
• Transactions. Multi-shard transactions need two-phase commit — supported by MongoDB and distributed SQL engines, at a real latency cost.
• Unique constraints. Uniqueness can only be enforced cheaply within a shard. A globally unique secondary field requires a separate lookup structure.
• Aggregations. COUNT, SUM and ORDER BY ... LIMIT require partial results from every shard merged by a coordinator.
• Backups. A consistent snapshot across shards needs coordination; per-shard backups taken at different moments do not restore to a single point in time.
• Schema changes. Every DDL statement must be applied to every shard, and must be safe to run while shards are at different versions.

Resharding

Consistent hashing with virtual nodes (Cassandra, Redis Cluster hash slots, MongoDB chunks) exists to make adding capacity incremental: only a fraction of keys move when a node joins, instead of nearly all of them as with plain hash(key) % node_count.

Before you shard

Sharding is usually the third option, not the first. Confirm you have exhausted the cheaper ones:

• Vertical scaling — a larger instance is often years of headroom for a fraction of the effort.
• Read replicas, if reads are the pressure.
• Archiving or partition-based retention, if size rather than throughput is the problem.
• Query and index work, if the load is self-inflicted. See Finding Slow Queries.
• Moving the analytical workload off the transactional database entirely.