All topics
40 / 66
Apache Cassandra

Backup & Restore: Snapshots & Incremental Backups

Learn how Cassandra's snapshot and incremental backup mechanisms work, and how to restore data from them.

Backing up Cassandra is unusually simple, because SSTables are never modified. A file that exists now will be byte-identical later or gone entirely.

A snapshot is like instantly stapling a copy of your current filing cabinet's folder list to a separate binder without duplicating the paper itself, since the pages don't actually move — but once you start shredding and replacing pages in the live cabinet, the binder's stapled copies become the only remaining originals. Incremental backups are like automatically stapling a copy of every new page into that binder the moment it's filed, so you can reconstruct anything added since your last full snapshot.

Key Concepts

1
So a backup is a set of file references, not a consistent dump.
2
A snapshot is hard links, so it is instant. nodetool snapshot creates hard links to the current SSTable files in a snapshots subdirectory:
nodetool snapshotsnapshots
3
    data/shop/orders/  nb-1-big-Data.db   <- the real file
    .../snapshots/tag/ nb-1-big-Data.db   <- another name for the same blocks
4
No copying happens. A terabyte snapshot takes a second and, at that moment, uses no extra space.
5
The space arrives later, and this catches teams out. Compaction merges SSTables and deletes the originals — except the snapshot still links them, so the blocks are not freed.
6
    day 0:  snapshot taken, 0 extra bytes
    day 7:  most of those SSTables compacted away
            -> the snapshot now holds the only reference
            -> its full size is now genuinely occupied
7
A snapshot nobody cleared is a very common cause of a node running out of disk. nodetool clearsnapshot and nodetool listsnapshots exist for this, and taking a snapshot should always come with a plan to remove it.
nodetool clearsnapshotnodetool listsnapshots
8
Incremental backups fill the gaps between snapshots. With the setting enabled, every SSTable is hard-linked into a backups directory as it is flushed. Snapshot weekly, keep the incrementals, and you can restore to a point between snapshots.
backups
9
Nothing deletes those either. That cleanup is your job.
10
Restoring one node. Stop it, remove the current SSTables for that table, copy the snapshot files back, start it, and run nodetool refresh so it picks them up.
nodetool refresh
11
Restoring a cluster is the hard part, and where interview answers usually stop too early. Three things make it more than repeating that for each node.
12
Snapshots are not synchronised. Each node snapshots at a slightly different moment, so the cluster-wide result is not a single point in time. Rows written during the window may be on some nodes and not others, and repair afterwards settles it.
13
Token ownership must match. Restore into a cluster with a different topology and the files contain data the node no longer owns. Either restore to matching tokens, or use sstableloader, which streams the files to whichever node owns them now.
sstableloader
14
Schema must exist first. The files are meaningless without their table definition, so back up the schema separately with DESCRIBE KEYSPACE and restore it before the data.
DESCRIBE KEYSPACE
15
And the one that is not a backup at all: replication. Three replicas protect you from a machine failing. They do not protect you from a DROP TABLE or a bad migration, which replicate perfectly to all three copies.
DROP TABLE
16
What the interviewer is probing.1. "What is a snapshot in Cassandra?" Probing: the mechanism. Stalls: "A copy." Moves up: hard links to the current SSTables, so it is near-instant and takes no extra space until compaction removes the originals.
17
2. "Why must snapshots be copied off the node?" Probing: the gap. Stalls: "They are safe." Moves up: they live on the same disk, so a node loss loses them too.
18
3. "How do you restore consistently across a cluster?" Probing: the difficulty. Stalls: "Restore each node." Moves up: restore matching snapshots on every node and run repair, since snapshots are not taken at a cluster-wide instant.
19
4. "What does incremental backup capture?" Probing: the detail. Stalls: "Everything." *Moves up:* new SSTables as they are flushed, which must be combined with a base snapshot and are not cleaned up automatically.
Use cases:backuprestoresnapshotsoperations