Performance Tuning: Bloom Filters, Key Cache & Row Cache

Understand the caching and probabilistic data structures Cassandra uses to accelerate reads, and how to tune them.

A read may need to check the memtable and every SSTable, because a row's columns can be scattered across all of them. Three structures keep that affordable.

The bloom filter is like a quick 'definitely not here' checklist you glance at before searching a filing cabinet, saving you from opening cabinets that certainly don't have what you need. The key cache is like remembering exactly which drawer and folder a frequently requested document lives in. The row cache is like keeping photocopies of your most-requested documents right on your desk so you never need to visit the cabinet at all — until someone updates that document, at which point your desk copy becomes stale and must be discarded.

Key Concepts

Bloom filters skip files without touching disk. Each SSTable has one. You ask "is this partition in this file?" and get one of two answers: definitely not, or probably yes.

    30 SSTables, bloom filters say:
      27 definitely not     -> skipped, zero disk access
       3 probably yes       -> 3 lookups (one may be a false positive)

Definitely-not is the valuable answer, and it is never wrong. A false positive costs one wasted lookup, never a wrong result.

bloom_filter_fp_chance is the dial: a lower false-positive rate skips more files and uses more memory. The default suits most tables; lowering it helps read-heavy tables with many SSTables, and raising it frees memory on tables that are rarely read.

bloom_filter_fp_chance

The key cache remembers where things are. Once you have located a partition inside an SSTable, the key cache stores that offset, so the next read jumps straight to it instead of consulting the on-disk index.

It stores offsets, not data — a few bytes per entry. That makes it the highest-value cache in Cassandra, and it is on by default. It should stay on.

The row cache stores whole rows, and usually should not be used. It caches the full row in memory, so a hit avoids disk entirely. Two problems make it a poor default.

It is invalidated by any write to that row. A row read often and written often is evicted constantly, so you pay to maintain a cache that rarely hits.

And it caches the entire partition, not the rows you asked for. On a wide partition — a device's whole day of readings — a query for the last 10 rows can pull megabytes into the cache.

It is off by default. Turn it on only for a small set of small, hot, rarely-written rows, and check the hit rate afterwards rather than assuming.

Measure, do not guess. nodetool tablestats reports bloom filter false-positive ratio, key cache hit rate, and SSTables per read. Those three numbers tell you which of the layers is failing you:

nodetool tablestats

    SSTables per read high  ->  compaction strategy is wrong for this workload
    bloom FP ratio high     ->  lower bloom_filter_fp_chance
    key cache hit rate low  ->  cache too small, or the access pattern is random

Tuning caches will not rescue a bad data model. If reads touch many partitions, or partitions are enormous, fix the schema first.

What the interviewer is probing.1. "What does a bloom filter cost and give you?" Probing: the trade. Stalls: "Nothing." Moves up: memory in exchange for skipping SSTables that cannot contain a key — a lower false positive rate costs more memory.

2. "When is the row cache harmful?" Probing: the caveat. Stalls: "Never." Moves up: for wide partitions or write-heavy tables — it caches whole partitions and is invalidated on every write.

3. "What does the key cache do?" Probing: the mechanism. Stalls: "Caches rows." Moves up: caches the partition index position, saving a disk seek — generally the safest cache to enable.

4. "What should you tune first?" Probing: the order. Stalls: "Caches." Moves up: the data model — no cache rescues a hot partition or a query needing ALLOW FILTERING.