Apache Cassandra
Anti-Entropy Repair
Learn how Cassandra reconciles replica divergence over time using repair, and why it's an essential operational task.
Replicas drift apart. A node was down for an hour, a mutation was dropped under load, a hint expired before it could be replayed.
Repair is like two branch offices periodically comparing summary checksums of their filing cabinets floor by floor and drawer by drawer, only pulling out and comparing actual documents in the specific drawers where the checksums don't match, instead of manually cross-checking every single page.
Key Concepts
1
Nothing in the normal write path notices. Repair is the process that finds the differences and fixes them.
2
The obvious approach is impossible. Comparing two replicas row by row across a few hundred gigabytes,
over the network, would take longer than the interval between repairs.
3
Merkle trees make the comparison small. Each replica hashes its data range into a tree: leaves hash
small sub-ranges, and each parent hashes its children.
4
replica A replica B
H(top) H(top) <- differ, so descend
/ \ / \
H(L) H(R) H(L) H(R) <- H(L) matches, stop there
H(R) differs, descend again5
If the root hashes match, the replicas are identical and nothing is transferred — the entire check is one
comparison. Where they differ, you descend only the branch that differs.
6
So the cost is proportional to how much data is wrong, not how much exists. That is what makes repair
practical.
7
Then only the differing ranges stream, and the newest timestamp wins per column, just as in a normal
read.
8
It is not automatic, and skipping it loses data. There are two reasons to run repair on a schedule, and
the second one surprises people.
9
The first is the obvious one: replicas converge, so reads at low consistency levels stop returning stale
values.
10
The second is tombstones. A tombstone may only be dropped after gc_grace_seconds, ten days by default,
and that window exists so every replica learns about the delete. Fail to repair a table within ten days and
a replica that missed the delete never learns of it — the tombstone elsewhere is collected, that replica's
copy is the only version left, and the deleted row comes back.
gc_grace_seconds
11
So the rule is: repair every table more often than gc_grace_seconds. Not "when things look wrong".
gc_grace_seconds
12
Full versus incremental. Full repair checks everything. Incremental repair marks data as repaired and
only examines what is new since last time, which is far cheaper and the sensible default for a large
cluster. Incremental repair had real bugs before Cassandra 4.0; on 4.0 and later it is the normal choice.
13
Do not run it by hand. Repair across a large cluster must be scheduled, staggered so nodes are not all
repairing at once, and monitored for failures. Reaper is the standard tool. A repair that has been quietly
failing for two weeks is indistinguishable from no repair at all — until a delete comes back.
14
What the interviewer is probing.1. "Why is repair necessary?" Probing: the model. Stalls: "To fix corruption." Moves up:
eventual consistency means replicas can diverge after failures or dropped hints; repair reconciles
them.
15
2. "How often must it run?" Probing: the rule. Stalls: "Occasionally." Moves up: within
gc_grace_seconds, or deleted data can be resurrected by a replica that never saw the tombstone.
16
3. "What is a Merkle tree used for?" Probing: the mechanism. Stalls: "Hashing." Moves up:
comparing replicas efficiently — only differing ranges are streamed, rather than whole datasets.
17
4. "What is incremental repair?" Probing: the variant. Stalls: "Faster repair." Moves up:
marking already-repaired data so later runs skip it, which is faster but has had correctness issues
in some versions.