Design Netflix / Video Streaming

hard

Scale: 200M users; 250M hours/day streamed; 100s of Tbps peak Storage: PB of encoded content; CDN cache distributed across thousands of PoPs Netflix, YouTube, Disney+

Case StudyStreamingCDN

Video streaming delivers tens of millions of concurrent streams globally with sub-second startup and smooth playback. Netflix is ~15% of internet traffic at peak. The system is dominated by transcoding (compute) and content delivery (bandwidth).

Scale200M users; 250M hours/day streamed; 100s of Tbps peak

StoragePB of encoded content; CDN cache distributed across thousands of PoPs

Key Concepts

1. Transcoding: the heavy compute. Master upload (~100 GB mezzanine) → transcode to a bitrate ladder of resolution × codec × container. Typical: 240p → 4K, H.264 / H.265 / AV1, HLS / DASH = 30+ outputs per title. Each split into 2-10 second segments. Netflix's per-shot encoding tunes bitrate per scene complexity, saving ~20% bandwidth. Compute is often Netflix's largest AWS bill.

1. Transcoding: the heavy compute.

2. Distribution: the dominant cost. 100M concurrent × 5 Mbps avg = 500 Tbps egress — larger than total transit between most continents. Netflix Open Connect embeds caching appliances inside ISPs: a Comcast subscriber's bits come from a Netflix box in Comcast's network. Zero transit, sub-50ms latency. Disney+, YouTube use commercial CDN (Akamai, CloudFront). Cache hit rate 95%+ thanks to heavy Zipf (top 1% = 50% of views).

2. Distribution: the dominant cost.

3. Playback: client-side ABR. Player downloads manifest (HLS .m3u8 or DASH .mpd) listing variants and segment URLs. Picks initial bitrate, downloads segments, measures bandwidth + buffer. Adaptive Bitrate (ABR) algorithms (BOLA, MPC, throughput-based) decide upshift vs downshift. Buffer-based smooth viewing > maximum quality. Goal: never rebuffer.

3. Playback: client-side ABR.

4. Recommendations and personalization. Nightly batch (Spark) computes per-user candidate sets via collaborative filtering + deep learning. Online reranking incorporates recent watches, time of day, device. Home page cached at edge with personalization via per-user signed URLs. Telemetry (start times, rebuffers, quality switches, completion) feeds the next training cycle — 1B+ events/day.

4. Recommendations and personalization.

5. Component map. Upload + S3. Transcoding cluster (Kafka job queue → encoders). Manifest service (per-title, per-region rules). CDN. Player + ABR. DRM license server (Widevine, FairPlay, PlayReady). Recommendation pipeline. Telemetry pipeline. Geo / catalog enforcement layer.

5. Component map.

High-level design

Ingest: master upload → object store (S3) → transcoding job.
Transcode: distributed workers consume Kafka jobs; produce bitrate ladder × codecs × HLS/DASH segments.
Distribute: segments pushed to CDN (Open Connect in-house, or commercial).
Playback: client fetches manifest → ABR algorithm picks bitrate → fetches segments → plays.
Recommendations: nightly batch (Spark) + online reranker.
Telemetry: client → Kafka → analytics + encoding optimization.
DRM: key server with per-session licenses.

Transcoding pipeline

Master: high-quality mezzanine file (~100 GB for a 2-hour movie).

Bitrate ladder: 235 kbps (240p) up to 16 Mbps (4K HDR), 10-15 variants.

Codecs: H.264 (universal), H.265 (better at high res, license cost), AV1 (royalty-free, expensive to encode).

Containers: HLS (Apple ecosystem), DASH (browsers, Android).

Segmentation: 2-10 second segments. Shorter = faster adaptation, more overhead. Longer = better compression.

Per-shot encoding (Netflix): per-scene bitrate optimization based on complexity.

Audio + subtitles encoded separately, multiplexed in playback.

CDN strategy

Open Connect (Netflix): caching appliances inside ISPs. Net savings = no transit, lowest latency, control over content placement.

Commercial CDN (Akamai, CloudFront, Cloudflare): pay per GB egress; broader reach but higher unit cost at scale.

Hybrid: hot titles on in-house cache; long tail on commercial.

Cache fill: pre-warm popular content (push), pull on miss for long tail.

Cache key: per-bitrate per-codec per-segment URL.

Hit rate target: 95%+ at edge. Heavy Zipf (top 1% = 50% of views) makes this achievable.

Adaptive bitrate (ABR)

Client measures bandwidth (recent segment download time) + buffer level (seconds of video ahead).

Throughput-based: pick bitrate based on recent average bandwidth.

Buffer-based (BOLA): pick bitrate based on buffer level; aggressive when buffer is healthy.

Hybrid (MPC, BOLA-MPC): model future buffer based on planned bitrate; pick to maximize quality without rebuffer.

Player tuning: startup bitrate (lower = faster start, lower initial quality), max upshift step, downshift sensitivity.

Components

Upload service + object store.
Transcoding cluster (Kafka job queue, encoder workers).
Manifest service (per-title, per-region rules).
CDN (push or pull with origin shield).
Playback client + ABR algorithm.
DRM license server.
Recommendation pipeline (batch + online).
Telemetry pipeline (start/stop/rebuffer/quality switch events).
Geo / DRM enforcement layer.

Trade-offs

Pre-transcode all bitrates: high storage cost, instant playback.

On-demand transcoding: less storage, latency at first view.

Per-shot encoding: 20% bandwidth saving, more encoder compute.

Open Connect: huge capex but lowest long-term unit cost.

Commercial CDN: faster to launch, higher per-GB cost.

Longer segments: better compression, worse ABR adaptation.