Apache Cassandra
Tunable Consistency & Hinted Handoff
Understand how hinted handoff and read repair work alongside consistency levels to maintain data integrity during transient failures.
Consistency levels decide how many replicas must answer. Three background mechanisms decide how the replicas that fell behind catch up.
Hinted handoff is like leaving a sticky note reminder for a coworker who stepped away, so you can hand them their messages the moment they're back at their desk instead of waiting for the next scheduled office-wide memo review. Read repair is like noticing two colleagues have different versions of a memo when you ask both, and quietly handing the older one the newer version on the spot.
Key Concepts
1
Without them, every small failure would need a full repair to fix.
2
Hinted handoff covers a replica that is briefly down. A write arrives for three replicas and one is
unreachable. The coordinator stores a hint — the mutation, plus who it was for — and keeps it:
3
write at QUORUM, RF=3
replica A ok
replica B ok -> quorum met, client gets success
replica C down -> coordinator holds a hint4
When C comes back, gossip tells the coordinator, and the hint is replayed. C catches up in seconds without
any repair.
5
The important detail for interviews: a hint does not count towards the consistency level. Writing at
QUORUM with only one live replica fails, even though the coordinator could store two hints. Hints are
catch-up, not a substitute for the replicas you asked for.
6
Hints also expire, three hours by default. A node down longer than that has gaps that only repair will
fill.
7
Read repair fixes divergence when someone notices. A read at QUORUM contacts two replicas and compares
what comes back. If they disagree, the newest wins, and Cassandra writes that value back to the stale
replica.
8
So frequently-read data heals itself as a side effect of being read. Rarely-read data does not, which is
the limitation — read repair only ever fixes rows somebody asked for.
9
Anti-entropy repair is the backstop. Scheduled, systematic, covering everything including the rows
nobody reads. The other two reduce how much it has to do; they do not replace it.
10
Why all three exist. Look at the gaps. Hints cover short outages, but expire. Read repair covers hot
data, but never touches cold data. Repair covers everything, but is expensive and periodic.
11
Together they mean a healthy cluster rarely has much to repair, and an unhealthy one still converges.
12
The tombstone reason again. Even a cluster where hinted handoff and read repair are working perfectly
must still run scheduled repair within gc_grace_seconds, because deleted rows are exactly the rows nobody
reads — so read repair will never encounter them.
gc_grace_seconds
13
What the interviewer is probing.1. "What is a hint?" Probing: the mechanism. Stalls: "A log entry." Moves up: when a
replica is down, the coordinator stores the write and replays it when the node returns.
14
2. "Why can hints not replace repair?" Probing: the limit. Stalls: "They can." Moves up:
they expire after a window and are not stored if the coordinator also fails, so divergence can
persist.
15
3. "How do you choose a consistency level?" Probing: the reasoning. Stalls: "Use QUORUM."
Moves up: from the data — R + W > RF where correctness demands it, lower where latency and
availability matter more.
16
4. "Can different queries use different levels?" Probing: the flexibility. Stalls: "No."
Moves up: yes, per statement — which is what tunable consistency means.