Tokens and Vnodes

A token is a position in the hash ring. Ownership of token ranges determines which node stores which partitions.

Single tokens versus virtual nodes

With single tokens, each node owns one large contiguous range. Adding a node splits one neighbour's range, so the cluster rebalances by moving data between two nodes only — and only that one neighbour's data.

With virtual nodes (vnodes), each node owns many small ranges scattered around the ring (num_tokens in cassandra.yaml). Adding a node takes a slice from many existing nodes at once, so streaming is parallel and the cluster stays balanced without manual token assignment.

# cassandra.yaml
num_tokens: 16
allocate_tokens_for_local_replication_factor: 3

Choosing num_tokens

The historical default of 256 balances well and has real costs: repair and streaming operate per range, so more ranges mean more overhead, and the probability that any given node pair shares a range approaches one — which increases the chance that losing two nodes loses a quorum for some partition.

Modern guidance favours a much lower value, commonly around 16, combined with allocate_tokens_for_local_replication_factor so token allocation is computed to balance ownership rather than assigned randomly.

Checking balance

nodetool status shop

Datacenter: eu-central
=====================
--  Address      Load       Tokens  Owns (effective)  Host ID   Rack
UN  10.20.1.10   412.3 GiB  16      33.4%             …         rack1
UN  10.20.1.11   408.1 GiB  16      33.3%             …         rack2
UN  10.20.1.12   415.7 GiB  16      33.3%             …         rack3

Owns (effective) should be near-equal across nodes in a datacenter. Load can differ more, because partition sizes vary — a large gap between nodes usually means hot partitions or an uneven partition key.

Racks and datacenters

Snitch configuration tells Cassandra the topology so replicas are placed in different racks:

endpoint_snitch: GossipingPropertyFileSnitch

# cassandra-rackdc.properties
dc=eu-central
rack=rack1

Adding and removing nodes

# Adding: start the node with auto_bootstrap enabled; it streams its ranges in.
nodetool netstats            # watch streaming progress
 
# Removing a live node: streams its data to the new owners first.
nodetool decommission
 
# Removing a dead node: reassigns its ranges without streaming from it.
nodetool removenode <host-id>