Back to System design
Design a Notification System
medium"Your order has shipped" needs to reach a user. It might go as a push notification, an email, an SMS, or a message inside the app — often several at once.
Scale10M+ notifications/min peak; sub-minute latency on transactional
StorageAudit log grows forever; preferences + suppression bounded
Key Concepts
1
Sending one message is easy. The system becomes hard because each channel fails differently, every user
has preferences, and the same message must never arrive twice.
2
One event, several messages. An order shipping produces a push notification, an email and an in-app
entry. Each goes to a different provider with its own rules, its own failures, and its own cost.
One event, several messages.
3
SMS costs real money per message. Push is free and silently disappears when the phone's token has expired.
Email is free and can quietly land in spam.
4
Preferences are the product, not a lookup. People turn channels off, set quiet hours, ask for a daily
summary instead of immediate alerts, and unsubscribe from categories.
Preferences are the product, not a lookup.
5
Each of those is a filter, and the order you apply them in matters. A password reset must ignore "no
marketing emails" and must ignore quiet hours. It must still respect someone who has unsubscribed
completely where the law requires it.
6
Never send the same thing twice. Retries and queues mean a message can be produced more than once. The
user does not see "a retry"; they see the same SMS at 2am, twice.
Never send the same thing twice.
7
So every notification gets a key made from the event, the channel and the recipient. Before sending, check
whether that key has been used. After sending, record it. A repeat becomes a no-op.
8
Each channel fails in its own way. Push tokens expire when someone reinstalls the app, and if you never
remove the dead ones your delivery rate quietly drops over months. SMS providers rate-limit and sometimes
drop messages without telling you. Email bounces in two ways: a hard bounce means the address does not
exist and you must stop, a soft bounce means try later.
Each channel fails in its own way.
9
Retrying a hard bounce damages your reputation with mail providers, and that damages delivery for every
email you send afterwards.
10
Protect the user from your own system. Twenty notifications in a minute is a reason to uninstall the
app. Cap how many a user can receive in an hour, combine related ones into a single message, and offer a
digest.
Protect the user from your own system.
11
What the interviewer is probing.
What the interviewer is probing.
12
1. "Design notifications." Probing: whether you see the steps. Stalls: "Call the push API when
something happens." Moves up: event, then preferences, then formatting, then per-provider sending with
retries and a place for permanent failures.
1. "Design notifications."
13
2. "The same SMS was sent twice." Probing: duplicate prevention. Stalls: "Fix the retry." *Moves
up:* a key from event plus channel plus recipient, checked and recorded around every send.
2. "The same SMS was sent twice."
14
3. "It is 3am for the user." Probing: preferences and timing. Stalls: "Send it anyway." Moves up:
quiet hours in their time zone, hold non-urgent messages until morning, and let security messages through.
3. "It is 3am for the user."
15
4. "Push delivery is dropping month by month." Probing: token hygiene. Stalls: "The provider is
flaky." Moves up: expired tokens pile up; delete them when the provider says they are invalid.
4. "Push delivery is dropping month by month."
16
5. "Emails are going to spam." Probing: whether you treat email as its own system. Stalls: "Change
the subject line." Moves up: stop sending to hard bounces immediately, set up the sender authentication
records, and watch the complaint rate.
5. "Emails are going to spam."
High-level design
Producers → NotificationRequest API.
Notification service → preference check, rate limit, dedup, template render.
Kafka topics per channel + priority (push-tx, push-bulk, email-tx, email-bulk, ...).
Per-channel worker pools → vendor API call with retries.
Vendor webhooks → status update → audit store.
Bounce handler → update suppression list.
Analytics → opens, clicks, conversion.
Components
- Notification API gateway.
- Preference service (per-user channel preferences).
- Suppression list (bounces, opt-outs).
- Rate limiter (per user, per template, per recipient).
- Template renderer (with localization, A/B variants).
- Channel workers (push, email, sms, in-app, slack).
- Vendor abstraction layer (multi-vendor with health-based routing).
- Audit + analytics store (Cassandra or ClickHouse).
- Webhook receiver for vendor status updates.
- Bounce / unsub processor.
Channel characteristics
Channel Latency Cost Failure mode
------------------------------------------------------------------
Push ~1 s free token expires silently;
delivery decays unnoticed
SMS 1-10 s per message carrier filtering, silent
drops, per-country rules
Email seconds ~free spam foldering; hard vs soft
bounce handling is mandatory
In-app instant free only seen when the app is open Retry policy per channel
Push 3 attempts, 1s / 10s / 60s; drop on invalid token
SMS 2 attempts, 30s apart; stop on permanent carrier error
(retrying costs money and annoys the user)
Email 5 attempts over hours for SOFT bounces; zero retries
for hard bounces, and suppress the address permanently
In-app write once to the store; it is a database row Cost check that changes designs
1M users x 1 SMS at 0.15 INR = 150,000 INR
Same notification by push = ~0
-> the preference default, and the fallback ordering, are a
finance decision as much as a UX one.Delivery pipeline
1. Event ingested (order.shipped) -> durable queue.
2. Expand to recipients (one user, or a group).
3. Apply preferences, in this order:
a. Hard suppression (unsubscribed, hard-bounced, invalid)
b. Category preferences (marketing off, transactional on)
c. Channel preferences (push yes, SMS no)
d. Quiet hours -> defer to a scheduled send, or digest
e. Rate limit per user (collapse or drop)
Security and legally required messages skip (b) and (d), never
(a).
4. Idempotency check: key = event_id + channel + recipient.
Already sent -> stop.
5. Render per channel (template + locale + timezone).
6. Dispatch to the provider adapter, with per-channel retry.
7. Record outcome; feed bounces and invalid tokens back into (a). Why the order matters
Applying rate limits before suppression wastes budget on messages
that will be dropped anyway; applying quiet hours before the
security exemption delays password resets until morning.Failure modes in production
Duplicate sends after a queue redelivery
Symptom: users receive the same message twice, hours apart.
Cause: at-least-once delivery with no idempotency record.
Fix: deterministic key checked and written before dispatch, with
a TTL longer than the maximum retry window.
Silent push decay
Symptom: delivery rate drifts from 95% to 60% over months.
Cause: expired tokens never pruned.
Fix: treat the provider's invalid-token response as authoritative
and delete immediately; audit token age. Hard bounces retried
Symptom: sender reputation falls; unrelated email starts landing
in spam.
Cause: retry policy applied uniformly to all failures.
Fix: distinguish hard from soft bounces; suppress hard bounces
permanently and alert on the rate. Notification storm from a batch job
Symptom: a backfill emits 200 events per user; phones buzz for
minutes.
Cause: no per-user rate limit or collapsing.
Fix: cap per user per hour, collapse related events into one
summary, and make batch jobs mark their events as digestible.