Explain Anti-Entropy Repair in Databases

32 / 66

Apache Cassandra

Anti-Entropy Repair

Learn how Cassandra reconciles replica divergence over time using repair, and why it's an essential operational task.

Because Cassandra favors availability over strict consistency by default, replicas can diverge over time — due to dropped mutations, hinted handoff failures, node downtime, or network issues. Anti-entropy repair is the maintenance process that reconciles these differences, ensuring all replicas eventually converge to the same data (fulfilling the "eventual" part of eventual consistency).

Repair is like two branch offices periodically comparing summary checksums of their filing cabinets floor by floor and drawer by drawer, only pulling out and comparing actual documents in the specific drawers where the checksums don't match, instead of manually cross-checking every single page.

Key Concepts

Repair works by comparing Merkle trees — hash trees representing the data in a partition range — between replicas. Rather than comparing every row byte-by-byte (impractical at scale), Merkle trees allow two replicas to quickly identify which sub-ranges of data differ by comparing hash values top-down, only exchanging actual data for the specific ranges where hashes mismatch.

Merkle trees

Repair is not automatic by default (aside from read repair, which is incidental and partial) — operators must run nodetool repair regularly, typically via a scheduled job, and this is critical for two reasons: fixing data divergence, and ensuring tombstones are properly propagated to all replicas before gc_grace_seconds expires (otherwise deleted data can resurrect).

tombstones are properly propagatednodetool repairgc_grace_seconds

Modern Cassandra deployments often use incremental repair (default since Cassandns 3.0/4.0 improvements) which only repairs data written since the last repair, or tools like Reaper (Cassandra Reaper) to automate and schedule repair across a cluster without manual intervention.

incremental repair