Apache Cassandra
Tombstones & TTL
Understand how Cassandra handles deletes and expirations via tombstones, and the operational risks of tombstone accumulation.
Deleting in Cassandra writes data. It does not remove any.
A tombstone is like sticking a 'DELETED' Post-it note over an entry in an old, unerasable ledger rather than tearing out the page — the note has to stay long enough for every branch office with a copy of the ledger to see it and cross out their own entry, before you're allowed to eventually shred that page.
Key Concepts
1
SSTables are immutable — once written, never edited. So a DELETE appends a marker saying "this is gone as
of this timestamp". That marker is a tombstone.
DELETE
2
Why the marker has to exist. Suppose the delete simply removed the row from the two replicas that were
reachable, and the third was down. When it returns, that replica has the row and the others do not.
3
Its copy looks like data and the others look like absence. Nothing distinguishes "never had it" from
"deleted it", so the old row spreads back to the others as if it were new. A tombstone is what makes the
delete a fact with a timestamp that can win a comparison.
4
TTL is the same thing on a timer. USING TTL 86400 marks the data to expire in a day. When it does,
the cell becomes a tombstone by itself:
USING TTL 86400
5
INSERT INTO sessions (id, data) VALUES (7, '...') USING TTL 3600;6
Perfect for sessions, caches and time-series data. Note what it means, though — a table where everything
expires is a table continuously generating tombstones.
7
Reads must step over every tombstone. To answer a query Cassandra reads the range and discards the
deleted entries. Deleted rows still cost work:
8
partition has 200 live rows and 90,000 tombstones
-> the read scans 90,200 entries to return 2009
This is the most common Cassandra performance failure. There are warning and failure thresholds
(tombstone_warn_threshold, tombstone_failure_threshold, 1,000 and 100,000 by default) and crossing the
second one makes the query fail outright.
tombstone_warn_thresholdtombstone_failure_threshold
10
Why they are not cleaned up immediately. gc_grace_seconds, ten days by default, is how long a
tombstone must survive before compaction may drop it.
gc_grace_seconds
11
That window exists so every replica learns about the delete. Drop the tombstone before a replica that
missed it has been repaired, and that replica still holds the original row — with nothing left to
contradict it. The row comes back. That is zombie data, and it is why the setting is measured in days.
12
The operational consequence: you must run repair on every table more often than gc_grace_seconds. If
repair falls behind that window, deleted data can return.
gc_grace_seconds
13
The modelling mistake behind most tombstone incidents. Using a Cassandra table as a queue — insert a
row, process it, delete it — means each partition accumulates tombstones where the work used to be, and
reads get slower every hour.
14
The fix is not tuning. It is TWCS with a TTL so whole SSTables expire and are dropped as files, or a
different tool. Also avoid deleting a whole collection or writing null, both of which create range
tombstones that are more expensive than they look.
null
15
What the interviewer is probing.1. "Why are tombstones a problem?" Probing: the read cost. Stalls: "They waste space."
Moves up: reads must scan them to know rows are deleted, so a partition with many tombstones gets
slower — and past a threshold the query fails.
16
2. "What is gc_grace_seconds for?" Probing: the correctness reason. Stalls: "Cleanup
timing." Moves up: it keeps tombstones long enough for repair to propagate the delete, or a node
that was down could resurrect deleted data.
17
3. "What anti-pattern generates tombstones fastest?" Probing: the modelling mistake. Stalls:
"Deleting rows." Moves up: using a partition as a queue — writing and deleting repeatedly leaves
the partition full of tombstones.
18
4. "How does TTL interact with this?" Probing: the connection. Stalls: "It avoids them."
Moves up: expired data also becomes tombstones, so a TTL-heavy table needs time-windowed
compaction to drop whole windows.