Sharding Strategies: Range, Hash, Directory, Geo

medium

Scale: Linear scale with shard count; cross-shard ops cost more Storage: Distributed across N shards + per-shard replicas Meta, Twitter, Uber

FundamentalsDatabasesSharding

Sharding splits a dataset across multiple physical nodes so the system can hold more data, handle more writes, and stay within per-node capacity. Each shard owns a subset of keys; a request routes to exactly one shard based on the shard key.

ScaleLinear scale with shard count; cross-shard ops cost more

StorageDistributed across N shards + per-shard replicas

Key Concepts

1. When to shard. Vertical scale runs out around 64-128 cores per node. Sharding is the answer when (a) dataset exceeds single-node storage (TB+), (b) write QPS exceeds single-node capacity (10K-50K writes/s for OLTP), or (c) a hot table contends even on replicas. Sharding too early is wasted ops cost; too late is an emergency.

1. When to shard.

2. Four strategies. Range: ordered key ranges per shard — cheap range scans, hot-tail risk on time-ordered keys. Hash: hash(key) mod N — even load but no range queries; naive mod-N moves all keys on rebalance. Consistent hashing with vnodes: minimal rebalance churn. Directory: lookup table maps key → shard — most flexible, extra hop, directory must be HA. Geo: shard by region for latency + compliance.

2. Four strategies.hash(key) mod N

3. The shard key is the most consequential decision. It must be present on every query (else: scatter-gather across all shards). It must have high cardinality (else: permanent hot shards). It must align with the dominant access pattern. Common picks: user_id (even load, no range scans), tenant_id (multi-tenant SaaS isolation), timestamp (time-series with TTL drops).

3. The shard key is the most consequential decision.

4. Operational pain points. Hot shards from celebrity users or mega-tenants — mitigate via split, replicate hot keys, or dedicated shard. Resharding without downtime: double-write + backfill + verify + cutover (Vitess, Citus automate this). Cross-shard transactions are expensive (2PC, sagas) — design the schema to keep transactional units within one shard.

4. Operational pain points.

5. In production. Pinterest: MySQL sharded by user_id. Slack: Postgres sharded by team_id (one team co-located → cheap queries). Discord migrated from MongoDB to Cassandra to ScyllaDB sharded by channel_id. Uber: schemaless directory mapping. Shopify: 'pods' — each pod is a complete shard with its own Postgres + Redis + worker pool.

5. In production.

Approach

Pick the shard key by aligning to the dominant access pattern. Most queries should hit exactly one shard.
Pick the strategy. Hash for even load and no range scans. Range for time-series and range queries. Directory for multi-tenant. Geo for compliance and latency.
Use consistent hashing with virtual nodes — naive mod-N is a footgun at scale.
Replicate within each shard. Sharding is for horizontal capacity; replicas are for HA.
Plan resharding from day one. Tools: Vitess, Citus, ShardingSphere, or roll your own with double-write + backfill.
Identify hot keys and plan mitigations: split, replicate, separate shard, or pre-aggregate.
Eliminate cross-shard transactions in the schema. Where unavoidable, use sagas or 2PC sparingly.
Build a routing layer: in-client (DBAL aware) or proxy (Vitess, ProxySQL). Proxy is easier to evolve; client is faster.

Strategies in depth

Range sharding: assign sorted key ranges to shards. Pro: range scans, locality. Con: hot tail when keys are time-ordered.

Hash sharding (mod-N): shard = hash(key) mod N. Pro: even distribution. Con: rebalance moves ~all keys.

Consistent hashing: hash keys + nodes onto a ring; vnodes for fine balance. Pro: minimal data movement on scale. Con: range scans hard.

Directory: lookup table keys → shard. Pro: most flexible; per-tenant moves are easy. Con: extra hop, directory HA required.

Geo: shard by region. Pro: compliance + latency. Con: cross-region queries hard; capacity skews by region.

Components

Shard map service / metadata store (etcd, ZooKeeper, custom).
Routing layer — in-client library or proxy (Vitess vtgate, ProxySQL).
Per-shard replica set for HA (3-5 replicas).
Rebalancer / splitter — splits hot shards, merges cold ones.
Backup / restore per shard; coordinated PITR across shards.
Cross-shard query layer if unavoidable (scatter-gather with merge).
Monitoring per shard — load, latency, replica lag, hot key detection.

Resharding playbook

1. Pick the new shard layout (e.g. split shard 5 into 5a + 5b).
2. Provision new shards.
3. Start double-writing: writes go to both old and new shards.
4. Backfill historical data from old to new.
5. Verify with shadow reads — read both, compare results, alert on mismatch.
6. Cut over reads to new shards.
7. Stop writes to old shards.
8. Decommission old shards after a safety window.
Tools that automate this: Vitess (MySQL), Citus (Postgres), DynamoDB auto-scaling, Bigtable range splits.

Trade-offs

Range: simple range scans, but hotspots common.

Hash: even load, no range scans, brutal rebalance with naive mod.

Consistent hashing: minimal rebalance churn; harder to reason about than range/directory.

Directory: most flexible; extra latency hop and operational cost (the directory itself needs HA).

Cross-shard transactions: expensive (2PC, sagas). Design the schema to avoid them whenever possible.

Sharding too early: extra ops cost with no benefit. Sharding too late: emergency under fire. Sweet spot is when you're at 50-70% of single-node capacity.

Real-world examples

Pinterest: MySQL sharded by user_id with manual shard splits and Vitess-style routing.
Slack: Postgres sharded by team_id; one team's data co-located for cheap queries.
Discord: migrated from MongoDB to Cassandra (then ScyllaDB) for message store; sharded by channel.
Uber: schemaless layer over MySQL with directory-based shard mapping.
Shopify: pods architecture — each pod is a complete shard with its own Postgres, Redis, etc.