Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Cassandraintermediate

Gossip Protocol

How Cassandra nodes learn about each other without a coordinator, and what to do when the cluster view disagrees.

2 min readIntermediateUpdated Edit this page
Apache CassandraScyllaDB

Gossip is how Cassandra maintains cluster membership without a configuration server. Once per second, each node exchanges state with a few others: node status, token ownership, schema version, load and heartbeat.

State converges in logarithmic time relative to cluster size, so a cluster of a hundred nodes learns about a change in seconds.

Seeds

Seed nodes are the contact points a starting node gossips with first. They are not special afterwards — a running cluster does not depend on them.

seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.20.1.10,10.20.1.11,10.20.2.10"

Failure detection

Cassandra uses a phi accrual failure detector: rather than a fixed timeout, it computes a suspicion level from the history of heartbeat intervals. A node is marked down when the suspicion exceeds phi_convict_threshold (8 by default).

Raise it on networks with variable latency — cloud environments and cross-region links — to reduce false positives. Lowering it makes failure detection faster and false convictions more likely.

States a node can be in

nodetool status combines two letters:

StatusMeaning
UNUp, Normal — healthy and serving
DNDown, Normal — unreachable, still owns ranges
UJUp, Joining — bootstrapping and streaming data in
ULUp, Leaving — decommissioning and streaming data out
UMUp, Moving — changing token assignment

A node in UJ for a long time is streaming; check nodetool netstats. A node stuck in UL has a decommission that has not completed.

When the cluster view disagrees

Different nodes reporting different nodetool status output means gossip state has not converged — usually a network partition, or a node that was removed improperly.

nodetool gossipinfo          # detailed state this node holds about every peer
nodetool describecluster     # schema versions; all nodes must agree

Operational notes

  • Keep clocks synchronised. Gossip state carries timestamps, and skew produces confusing membership behaviour on top of the data correctness problems it already causes.
  • Do not restart several nodes at once. Gossip needs time to converge, and simultaneous restarts can leave ranges without an available replica.
  • After any topology change, verify with nodetool status from more than one node — agreement across nodes is the actual success criterion.