Design Dropbox / Google Drive

Client: chunker + sync engine + local SQLite metadata cache.
Upload: chunk file → hash → query server for missing → upload missing chunks → commit metadata.
Download: poll/listen for namespace changes → fetch metadata → download missing chunks.
Sharing: ACL on namespace; commits emit events to all members.
Versioning: old metadata revisions retained; chunks reference-counted.
Search: async indexer reads metadata + extracts content for indexable file types.

• Client SDK (chunker, sync engine, conflict resolver).
• Metadata service (sharded by namespace/user_id); strong consistency.
• Block store (S3-class object storage with replication + erasure coding).
• Block index (chunk hash → blob location + reference count).
• Notification service (WebSocket / long poll) for change events.
• Search indexer for content search.
• Audit + versioning store.
• ACL service for sharing permissions.
• Sync conflict detector.

  Users                                         100,000,000
  Average stored per user                              10 GB
  Block size                                             4 MB
  ---------------------------------------------------------------
  Logical data      100M x 10 GB        =               1 EB
  After dedup and compression (assume 30% saving)
                                        =             700 PB

  Blocks per user   10 GB / 4 MB        =         2,560 blocks
  Metadata row per block
    block hash 32 B + offset 8 B + version id 8 B + flags 4 B
                                        =            ~52 bytes
  Per user          2,560 x 52 B        =              133 KB
  All users         100M x 133 KB       =             13.3 TB
                                        (metadata alone, before
                                         indexes and versions)

  The small-file case that breaks naive designs
    A user with 500,000 files of 20 KB
    Data                                =               10 GB
    Blocks (one partial block per file) =             500,000
    Metadata rows                       =             500,000
    -> same data volume as the average user, 200x the metadata.

Consequences:
- Dedup pays for itself at this scale; the same OS installer, the same
company deck, stored once.
- 13 TB of metadata is a sharded database problem with real indexes, not
an afterthought attached to the block store.
- Per-user sharding of metadata keeps a single user's listing fast, which
is what they actually notice.

  Client state
    last_seen_version (a cursor, not a timestamp -- clocks disagree)

  On reconnect
    1. GET /delta?cursor=<last_seen_version>
    2. Server returns metadata changes since the cursor, plus a new cursor.
    3. Client diffs locally: which block hashes does it already have?
    4. Client fetches only missing blocks from object storage (signed URL,
       so the bytes never pass through the application tier).
    5. Client commits locally and advances the cursor.

  On local change
    1. Chunk the file, hash each block.
    2. Ask the server which hashes are unknown (this is where dedup pays:
       usually most are known).
    3. Upload the unknown blocks.
    4. Commit a new version referencing the full block list.

  While online
    A notification channel pushes "something changed" so step 1 happens
    immediately rather than on a timer. The cursor makes the notification
    advisory -- losing one costs latency, not correctness.

  Why a cursor rather than a timestamp
    Two servers writing metadata have different clocks. A monotonic version
    per user is unambiguous and survives clock skew, leap seconds and
    backwards NTP steps.

Rename storm
Symptom: renaming a folder with 50,000 files saturates sync for hours.
Cause: rename modelled as delete + create per file.
Fix: model the move as a single metadata operation on the subtree;
blocks never move.

  Partial-block churn
    Symptom: an append-only log file re-uploads its last block constantly.
    Cause: fixed block boundaries meeting appends.
    Fix: content-defined chunking (rolling hash) for files that change by
    insertion, which keeps boundaries stable.

  Dedup across tenants leaking existence
    Symptom: an attacker learns whether a file exists by timing the "do you
    already have this hash?" call.
    Cause: global dedup exposed to the client.
    Fix: dedup per tenant, or require proof of possession before accepting
    a hash match.

  Reference count drift
    Symptom: storage grows despite deletions; orphaned blocks accumulate.
    Cause: counts updated in a separate transaction from version commits.
    Fix: treat reclamation as a periodic mark-and-sweep over metadata
    rather than trusting incremental counts.