Driver Policies: Load Balancing, Retry & Reconnection

Understand how Cassandra client drivers use pluggable policies to route requests, handle failures, and recover from outages.

The driver decides which node to talk to, what to do when that fails, and when to try a dead node again. Those defaults matter more than most application code.

The load balancing policy is like a dispatcher who always tries to route calls to the nearest available support office first. The retry policy is like a rulebook telling the dispatcher exactly when it's safe to redial after a dropped call versus when to just tell the customer it failed. The reconnection policy is like waiting progressively longer between redial attempts to an office that stopped answering, so you don't flood a struggling office with call attempts the moment it goes quiet.

Key Concepts

The load balancing policy picks the coordinator. It should prefer the local datacentre:

    datastax-java-driver.basic.load-balancing-policy {
      local-datacenter = mumbai
    }

Get this name wrong and the driver treats every node as remote — or picks coordinators in Virginia, adding a cross-ocean hop to every request that then still has to come back. It must match the snitch's datacentre name exactly.

Good drivers also do token-aware routing: they compute the partition key's token themselves and send the request straight to a replica that holds the data. That removes one network hop, since the coordinator is now also a replica. It works only when the driver can see the key, which means using bound parameters rather than string-concatenated CQL.

The retry policy is where correctness is decided. A timeout does not tell you whether the write happened. The request may have succeeded with the acknowledgement lost.

So retrying is safe only when the operation is idempotent:

    UPDATE users SET email = 'a@x.com'   -- safe, same result twice
    UPDATE counts SET n = n + 1          -- not safe, counts twice
    INSERT ... IF NOT EXISTS             -- not safe to blindly retry

The driver cannot tell these apart, so you mark statements idempotent and it retries only those. Leaving that flag wrong in either direction is a real bug: too eager duplicates counter increments, too cautious fails requests that would have succeeded on another node.

The reconnection policy stops you finishing off a struggling node. When a node is marked down, the driver waits before trying again, doubling the interval up to a maximum.

    attempt after 1s, 2s, 4s, 8s ... up to 60s

Without backoff, every client reconnects continuously to a node that is already overloaded, which is how a recovering node gets pushed back down. Exponential backoff gives it room to finish starting.

Two more settings worth naming. A speculative execution policy sends the same read to a second replica when the first is slow, trading extra load for a better tail latency. And request timeouts should be shorter than the user-facing timeout above them, so a slow node fails fast enough to be retried elsewhere rather than consuming the whole budget.

What the interviewer is probing.1. "What does a token-aware policy do?" Probing: the optimisation. Stalls: "Balances load." Moves up: it sends the request directly to a replica for that partition, removing a coordinator hop.

2. "Why is a datacentre-aware policy important?" Probing: the consequence. Stalls: "Tidiness." Moves up: without it, requests cross regions, adding latency and cost.

3. "What is the risk of an aggressive retry policy?" Probing: the hazard. Stalls: "None." Moves up: retrying a non-idempotent statement can apply it twice — counters and lightweight transactions especially.

4. "What should the driver be configured with for timeouts?" Probing: the practical detail. Stalls: "The defaults." Moves up: timeouts shorter than the server-side ones and a speculative execution policy where tail latency matters.