Back to System design

Design Dropbox / Google Drive

hard
Scale: 500M users; 10K-100K uploads/s Storage: EB-scale; dedup saves 20-50% depending on policy Dropbox, Google, Microsoft
Case StudyStorageSync

A file sync service like Dropbox, Google Drive, OneDrive, iCloud lets users save once and have files appear everywhere, share with collaborators, and recover old versions. The architecture problems are chunking, deduplication, multi-device sync, and conflict resolution.

Scale500M users; 10K-100K uploads/s
StorageEB-scale; dedup saves 20-50% depending on policy

Key Concepts

1
1. Content-addressed chunking. Files split into ~4 MB chunks; each chunk hashed (SHA-256). File = ordered list of chunk hashes + metadata. Editing 2 MB of a 100 MB file uploads only the 1-2 changed chunks. Identical chunks dedupe — Dropbox once estimated 50% global savings; now per-user only for privacy.
1. Content-addressed chunking.
2
2. Two separate storage systems. Block storage keyed by hash — content-addressable, append-only, S3-class with replication and erasure coding. Petabyte scale. Metadata service maps namespace + path → list of chunk hashes + version. Smaller (a few KB per file) but transactionally demanding — sharded SQL or NewSQL; strong consistency.
2. Two separate storage systems.namespace + path → list of chunk hashes + version
3
3. The sync protocol. Client computes block list locally. Asks server which blocks it lacks. Uploads missing blocks (parallel HTTP PUT). Calls commit: server validates blocks exist, atomically updates metadata. On success, server emits ChangeEvent to namespace notification channel (WebSocket / long poll). Other devices on that namespace pull new metadata, download missing chunks.
3. The sync protocol.
4
4. Conflict resolution. Server is source of truth. Commit uses CAS on parent version. Loser sees stale parent → server returns current. Auto-merge if changes don't overlap (rare for binary); otherwise create foo (conflicted copy from device X).ext with the loser's content. Real text/spreadsheet merging is out of scope — that's collaborative editor territory.
4. Conflict resolution.foo (conflicted copy from device X).ext
5
5. In production. Block dedup per user (privacy). Versioning: keep historical metadata; chunks reference-counted; old chunks GC'd. Sharing: ACL on namespace; ChangeEvents fan out to all members. Mobile clients use selective sync (storage limits). Convergent encryption (hash → key) enables E2EE + dedup but vulnerable to confirmation attacks. References: Dropbox Magic Pocket, Box, Google Drive.
5. In production.

Related Problems

Design a URL Shortener (TinyURL / bit.ly)mediumDesign Twitter / News FeedhardDesign WhatsApp / Chat SystemhardDesign Uber / Ride-HailinghardDesign Netflix / Video Streaminghard

High-level design

Client: chunker + sync engine + local SQLite metadata cache.
Upload: chunk file → hash → query server for missing → upload missing chunks → commit metadata.
Download: poll/listen for namespace changes → fetch metadata → download missing chunks.
Sharing: ACL on namespace; commits emit events to all members.
Versioning: old metadata revisions retained; chunks reference-counted.
Search: async indexer reads metadata + extracts content for indexable file types.

Components

  • Client SDK (chunker, sync engine, conflict resolver).
  • Metadata service (sharded by namespace/user_id); strong consistency.
  • Block store (S3-class object storage with replication + erasure coding).
  • Block index (chunk hash → blob location + reference count).
  • Notification service (WebSocket / long poll) for change events.
  • Search indexer for content search.
  • Audit + versioning store.
  • ACL service for sharing permissions.
  • Sync conflict detector.

Sync protocol in detail

  1. Client computes block list (4 MB chunks, SHA-256 hashes).
  2. Client sends list to server: which blocks are missing?
  3. Server responds with the subset client needs to upload.
  4. Client uploads missing blocks (parallel HTTP PUT to block servers).
  5. Client commits: send {path, parent_version, new_block_list, mtime} to metadata service.
  6. Server validates blocks exist and parent_version matches; if conflict, return current.
  7. On success, server emits ChangeEvent to namespace notification channel.
  8. Other clients receive event, pull new metadata, fetch missing blocks.

Conflict handling

First write wins on the metadata commit (CAS on version).

Loser sees stale parent_version → server returns current state.

Client option: auto-merge if changes don't overlap (rare for binary files; common for text).

Otherwise: create 'foo (conflicted copy from device X).ext' with the loser's content; both files now in tree.

User resolves manually. UI surfaces conflicts prominently.

Dedup and encryption

Per-user dedup: chunks dedupe across user's own files. No privacy concern.

Global dedup: chunks dedupe across all users. Privacy concern (Dropbox abandoned for this reason).

Client-side encryption (E2EE): server can't dedupe at all because identical plaintext maps to different ciphertext.

Convergent encryption: hash plaintext → key → encrypt; identical plaintext maps to identical ciphertext. Enables E2EE + dedup but vulnerable to confirmation attacks.

Trade-offs

Small chunks: more dedup, more metadata.

Large chunks: less metadata, less dedup, less efficient incremental sync.

Push notifications: low-latency sync; needs persistent connections.

Polling: simpler, higher latency, more cost.

Per-user vs global dedup: privacy vs storage savings.

Versioning forever vs N days vs paid-only: cost vs UX.

Mobile clients: sync only flagged files (storage limits); selective sync paradigm.