Apache Cassandra
Gossip Protocol & Failure Detection
Understand how Cassandra nodes discover cluster state and detect failures without a central coordinator.
No node is in charge, so no node can be asked who is alive. Cassandra has the nodes tell each other, continuously.
Gossip protocol works like office rumors — you don't need a company-wide announcement system; you just tell a couple of coworkers, they tell a couple more, and within minutes everyone knows, without anyone acting as a central broadcaster.
Key Concepts
1
How gossip spreads. Once a second, every node picks up to three peers at random and exchanges what it
knows: who is up, how loaded they are, which schema version they run, which tokens they own.
2
Each of those peers does the same a second later. Knowledge spreads by multiplying:
3
second 1: 1 node knows
second 2: 3 know
second 3: 9 know
second 4: 27 know4
A hundred-node cluster learns anything in about five rounds. No coordinator, no broadcast storm — each node
sends three small messages a second regardless of cluster size.
5
Detecting death is harder than it looks. The obvious rule is "no heartbeat for 5 seconds means dead".
That rule is wrong in both directions.
6
On a congested network a healthy node misses the deadline and gets declared dead, so its data is
redistributed for nothing. On a fast local network a node can be dead for 5 full seconds while requests are
still sent to it.
7
So Cassandra measures suspicion instead of applying a deadline. The Phi Accrual Failure Detector keeps
the recent history of gap sizes between heartbeats from each peer, and asks: given that history, how
surprising is the current silence?
8
The answer is a number, phi, that rises the longer the silence runs relative to what is normal *for that
peer*. On a link where heartbeats normally arrive every 1.1 seconds, a 3-second gap is alarming. On a link
that routinely varies between 1 and 8 seconds, the same gap is unremarkable.
9
The threshold is configurable (phi of 8 by default), so you tune one number instead of guessing a timeout,
and the detector adapts as network conditions change.
10
Each node decides for itself. A node marked DOWN is marked down by the node that noticed. There is
no vote and no cluster-wide agreement.
11
So during a network partition the two sides genuinely hold different views of who is alive, and both keep
serving. That is the availability trade, visible in the machinery.
12
Gossip carries more than liveness. Schema changes propagate this way, which is why a new table appears
on every node within seconds and why a node that was offline during a schema change catches up on rejoining.
Token ownership moves the same way during scaling. And a joining node needs only a seed node's address —
one contact is enough to learn the entire cluster.
13
What the interviewer is probing.1. "What does gossip do?" Probing: the purpose. Stalls: "Sends data." Moves up: nodes
exchange state about themselves and others a few times a second, so membership and liveness
propagate without a coordinator.
14
2. "How does Cassandra decide a node is down?" Probing: the mechanism. Stalls: "A timeout."
Moves up: the phi accrual failure detector, which produces a suspicion level from the history of
heartbeat intervals rather than a fixed threshold.
15
3. "Why is that better than a fixed timeout?" Probing: the benefit. Stalls: "It is not."
Moves up: it adapts to network conditions, so a temporarily slow link does not immediately mark a
healthy node as down.
16
4. "What are seed nodes for?" Probing: the detail. Stalls: "They are masters." Moves up:
bootstrapping gossip for a joining node — they have no special role afterwards.