Design WhatsApp / Chat System

hard

Scale: 100B msgs/day → ~1.2M msg/s peak globally Storage: ~300 B/msg × 100B = 30 TB/day (encrypted) Meta (WhatsApp), Signal, Telegram

Case StudyRealtimeMessaging

A real-time chat system serves 1:1 and group messaging with persistence, presence, delivery receipts, push for offline users, and (often) end-to-end encryption. WhatsApp (3B users, 100B messages/day), Signal, iMessage, Discord all share the same shape.

Scale100B msgs/day → ~1.2M msg/s peak globally

Storage~300 B/msg × 100B = 30 TB/day (encrypted)

Key Concepts

1. The connection layer. Clients open a long-lived connection — WebSocket (browser, modern mobile) or MQTT / XMPP (legacy, IoT) — to a gateway. The gateway is stateful: it knows which user owns which socket. Routing requires either sticky load balancing (hash by user_id) or a presence service tracking user_id → gateway_id. A typical gateway holds 100K-1M connections per process via epoll/kqueue.

1. The connection layer.user_id → gateway_id

2. The message flow. A sends to B. A's client → A's gateway → message service. Message persists to A's outbox + B's inbox (Cassandra wide row keyed by user_id, clustered by timestamp). Presence check: B online → push directly to B's gateway → socket. B offline → push notification via APNs (iOS) or FCM (Android). When B reconnects, client pulls messages since last seen offset.

2. The message flow.

3. Group chats and supergroups. Server fans out to every member's inbox — bounded by group size cap (WhatsApp 256, Signal 1000). Each member's device tracks its own delivery offset. Supergroups (Telegram-style 200K+) use a different model — broadcast channel, not fanout. The cost grows linearly with group size; cap aggressively.

3. Group chats and supergroups.

4. End-to-end encryption (Signal Protocol). X3DH key agreement: each user publishes identity key + signed prekey + onetime prekeys; sender derives shared secret. Double Ratchet: per-message keys with forward secrecy and post-compromise security. Server stores ciphertext blobs — cannot read content. Multi-device complicates: per-device key pair; server fans out the same ciphertext to N device queues per recipient.

4. End-to-end encryption (Signal Protocol).

5. Production essentials. Storage: ~300 B/msg × 100B = 30 TB/day. Concurrency: 200M+ live connections globally; 100K-1M per gateway → 200-2000 gateways. WhatsApp famously ran a few dozen Erlang servers per million users — Erlang/Elixir's process model is well-suited; Go and Rust also common. Push batching (APNs / FCM) is essential to stay under per-app per-second limits.

5. Production essentials.

High-level design

Connection: long-lived WebSocket/XMPP to gateway; sticky by user_id.
Send: client → gateway → message service → persist (sender outbox + recipient inbox) → presence → online? push to gateway : queue push notification.
Receive: client opens socket → server delivers buffered messages → client acks → server marks delivered.
Groups: fanout to per-recipient inboxes (bounded by group size cap).
E2EE: client-side encryption with Signal Protocol; server stores ciphertext only.
Media: encrypted blobs in S3-class storage.

Components

WebSocket gateway (stateful; user → connection routing). Erlang/Elixir famous fit; Go and Rust also common.
Presence service (Redis with TTL heartbeat).
Message service + per-user inbox (Cassandra wide row by user_id, clustered by timestamp).
Push gateway (APNs, FCM with batching and retry).
Media service (encrypted blobs in S3 + CDN for distribution).
Group metadata service (members, admins, settings).
E2EE key server (prekey bundles, identity keys, signed prekeys for X3DH).
Anti-abuse service (spam, content moderation if not E2EE).

Routing and presence

Stateful gateways: hash by user_id for sticky LB. Or: presence service maps user → gateway; sender looks it up.

Presence with Redis: per-user key with current gateway ID, refreshed by heartbeat. TTL-based cleanup.

Gateway failure: connections drop, client reconnects to new gateway (sticky changes), inbox is canonical so no data loss.

Multi-region: presence can be per-region with cross-region forwarding when sender and recipient are in different regions.

E2EE: Signal Protocol

X3DH key agreement: each user publishes identity key + signed prekey + onetime prekeys; sender combines to derive shared secret.

Double Ratchet: per-message key derived from chain key + DH ratchet; forward secrecy + post-compromise security.

Server stores: ciphertext blobs, prekey bundles. Cannot read messages.

Multi-device: per-device key pair; sender encrypts once per recipient device; server fans out ciphertext.

Trade-offs

Stateful gateways: efficient routing, but failover requires reconnects; need consistent hash or presence service.

Group fanout: cost grows with group size; cap or use special supergroup model (Telegram-style channels).

E2EE: privacy, but no server-side abuse / search / moderation.

Store-and-forward vs pure routing: store gives offline + multi-device sync but uses more disk.

WebSocket vs MQTT: MQTT is lighter, better for poor networks; WebSocket more ubiquitous.

Capacity sketch

100B msgs/day = ~1.2M msg/s peak.
300 B/msg × 100B = 30 TB/day storage.
200M concurrent connections globally; 100K-1M per gateway → ~200-2000 gateways.
Inbox queries are point reads in Cassandra; ~1 ms p99 with sharded cluster.
APNs / FCM: batched (multiple messages per push) to stay under per-app per-second limits.