Apache Cassandra
Gossip Protocol & Failure Detection
Understand how Cassandra nodes discover cluster state and detect failures without a central coordinator.
Cassandra uses a gossip protocol for peer-to-peer communication of cluster state, including node liveness, load, schema version, and token ownership. Every second, each node exchanges state information with up to three random peers, and information propagates exponentially across the cluster — similar to how rumors spread in a social network.
Gossip protocol works like office rumors — you don't need a company-wide announcement system; you just tell a couple of coworkers, they tell a couple more, and within minutes everyone knows, without anyone acting as a central broadcaster.
Key Concepts
1
Because there's no central coordinator to track node health, Cassandra uses an accrual failure detector (specifically, the Phi Accrual Failure Detector) instead of simple heartbeat timeouts. Rather than binary up/down decisions, it calculates a suspicion level (phi value) based on historical inter-arrival times of heartbeats, adapting to network conditions dynamically rather than using a fixed timeout.
accrual failure detectorPhi Accrual Failure Detector
2
When phi exceeds a configurable threshold, the node is marked as DOWN locally by the observing node — but this determination isn't necessarily agreed upon cluster-wide immediately, since each node makes its own independent judgment based on gossip data it has received.
DOWN
3
Gossip also underpins schema propagation (new tables/keyspaces), token ownership changes during scaling operations, and seed node discovery when a new node joins the cluster.