Back to System design
Design Netflix / Video Streaming
hardTen million people press play at 8pm. Each one needs a steady stream of video for the next two hours, and any interruption is obvious.
Scale200M users; 250M hours/day streamed; 100s of Tbps peak
StoragePB of encoded content; CDN cache distributed across thousands of PoPs
Key Concepts
1
Start with how much data that is, because the answer rules out most designs before you draw anything.
2
The bandwidth sum. A 1080p stream needs about 5 megabits per second. Ten million of them is 50
terabits per second.
The bandwidth sum.50
terabits per second
3
No single data centre can send that. Not with a bigger server, not with a faster network card. The video
has to already be close to the viewer before they press play.
4
So streaming companies put storage boxes inside the internet providers' own networks. When you press play,
the video comes from a machine in your ISP's building, a few milliseconds away, and never crosses an
expensive long-distance link.
5
Prepare the video once, serve it millions of times. Every film is converted ahead of time into many
versions: several resolutions, each at several quality levels, in several formats for different devices.
Each version is then cut into chunks of a few seconds.
Prepare the video once, serve it millions of times.
6
A two-hour film becomes tens of thousands of small files. That sounds wasteful and is not: it is what lets
any device play it, and it is what makes the next part possible.
7
The player picks the quality, not the server. Your device downloads a small list of available versions.
It measures how fast the chunks are arriving and how much it has buffered, then chooses which version to
request next.
The player picks the quality, not the server.
8
This is why video keeps playing when you walk into a lift. The player drops to a lower quality, keeps the
buffer alive, and climbs back up when the signal returns. The server does nothing clever.
9
The failure to mention is flapping: a player that jumps between qualities every few seconds because it
reacts to every small change. The fix is to require a sustained improvement before moving up, while
dropping down immediately.
10
Send popular titles out before anyone asks. A normal cache fills up as people request things. That
means the first viewers of a new release all miss, and they all wait while the origin is hammered by
hundreds of edge servers at once.
Send popular titles out before anyone asks.
11
Because you know the release date, you can push the files out overnight instead. By midnight every edge
already has it.
12
No edge holds everything. The full catalogue is about 10 petabytes for one copy. Each edge box holds
the titles its own region actually watches — a couple of thousand of them. Anything rarer is fetched from
a regional cache, and the extra fraction of a second is invisible next to the player's own buffering.
No edge holds everything.
13
What the interviewer is probing.
What the interviewer is probing.
14
1. "Design Netflix." Probing: whether you compute the traffic first. Stalls: listing services —
catalogue, users, recommendations. Moves up: 10 million viewers times 5 Mbps is 50 Tbps, so the video
must sit inside ISP networks.
1. "Design Netflix."
15
2. "How does the player choose quality?" Probing: who decides. Stalls: "The server detects the
connection." Moves up: the player measures its own buffer and picks the next chunk; plus flapping and
the fix.
2. "How does the player choose quality?"
16
3. "A new season launches at midnight." Probing: filling a cache versus pre-loading it. Stalls:
"The CDN caches it after the first request." Moves up: that means hundreds of edges all fetching at
once; push it out the night before.
3. "A new season launches at midnight."
17
4. "How much storage at the edge?" Probing: everything versus the popular part. Stalls: "The whole
catalogue." Moves up: 10 PB for one copy, so each edge keeps the couple of thousand titles its region
watches.
4. "How much storage at the edge?"
18
5. "Why cut the video into chunks?" Probing: why not one big file. Stalls: "It would work." *Moves
up:* chunks can be cached separately and let the player switch quality mid-stream; one file locks you into
one quality.
5. "Why cut the video into chunks?"
High-level design
Ingest: master upload → object store (S3) → transcoding job.
Transcode: distributed workers consume Kafka jobs; produce bitrate ladder × codecs × HLS/DASH segments.
Distribute: segments pushed to CDN (Open Connect in-house, or commercial).
Playback: client fetches manifest → ABR algorithm picks bitrate → fetches segments → plays.
Recommendations: nightly batch (Spark) + online reranker.
Telemetry: client → Kafka → analytics + encoding optimization.
DRM: key server with per-session licenses.
Components
- Upload service + object store.
- Transcoding cluster (Kafka job queue, encoder workers).
- Manifest service (per-title, per-region rules).
- CDN (push or pull with origin shield).
- Playback client + ABR algorithm.
- DRM license server.
- Recommendation pipeline (batch + online).
- Telemetry pipeline (start/stop/rebuffer/quality switch events).
- Geo / DRM enforcement layer.
Bandwidth and storage, worked
Concurrent streams at peak 10,000,000
Bitrate, 1080p 5 Mbps
-------------------------------------------------------------
Aggregate egress 10M x 5 Mbps = 50 Tbps
If 20% watch 4K at 20 Mbps
8M x 5 Mbps + 2M x 20 Mbps = 80 Tbps A 100 Gbps edge appliance serves
100 Gbps / 5 Mbps = 20,000 streams
Appliances for 10M streams = 500
(before redundancy, and before the 4K mix above) Catalogue storage, one full copy
Titles 100,000
Ladder per title (all renditions, all codecs) ~100 GB
100,000 x 100 GB = 10 PB Edge site, head of the distribution
Titles covering ~90% of regional views ~2,000
2,000 x 100 GB = 200 TB
(fits an appliance) Segment count for one 2-hour title
Renditions x codecs 10 x 3 = 30
Segments at 4 s (7,200 / 4) = 1,800
Files 30 x 1,800 = 54,000Consequences:
- 50 Tbps is why appliances sit inside ISPs; transit at that volume is
the dominant cost of the business.
- 10 PB for one copy is why edges hold the head, not the catalogue.
- 54,000 files per title is why the transcode pipeline and object naming
scheme get design attention that the "app" never needs.
Why pre-position instead of cache
Demand-filled cache, new release at 00:00
First viewer per edge: miss -> origin fetch.
500 edges x 30 renditions = 15,000 origin fetches
...in the first seconds, all for the same title, while the whole
audience waits on the slowest of them. Pre-positioned, pushed overnight
Origin fetches at release time 0
First viewer is served from local storage. What decides placement
- Predicted popularity per region (past viewing of similar titles,
pre-release engagement).
- Language and subtitle demand: a title's Hindi audio track belongs in
some regions and not others.
- Appliance free space: eviction is by predicted future views, not by
recency, because recency is a poor predictor on a release schedule. What still needs demand-filling
The long tail. A 2009 documentary watched twice a month per region is
not worth pushing; it comes from a regional cache on request, and the
extra 40 ms of start-up is invisible next to the player's own buffering.Failure modes in production
Bitrate oscillation
Symptom: quality flips every few seconds; viewers report flicker.
Cause: the player upgrades on a brief throughput spike, then starves.
Fix: hysteresis -- require sustained headroom and a minimum buffer
before stepping up; step down immediately.
Premiere thundering herd on the long tail
Symptom: a surprise hit that was not pre-positioned saturates regional
caches.
Cause: placement driven only by predicted popularity.
Fix: real-time promotion -- when an unplaced title crosses a request
threshold, push it to edges mid-event rather than serving every miss
from origin. Appliance disk failure during peak
Symptom: one ISP's subscribers see buffering while the rest are fine.
Cause: single copy of the head at that site.
Fix: two appliances per site with overlapping heads, and failover to
the regional cache rather than to origin. Codec mismatch on old devices
Symptom: playback fails silently on a subset of TVs.
Cause: manifest advertises a rendition the device cannot decode.
Fix: device capability negotiation before manifest generation, and keep
an H.264 rung in the ladder long after it is inefficient.