Back to System design

Design a Web Crawler

hard
Scale: 1B pages, weekly refresh → ~1700 pages/s sustained Storage: Compressed HTML ~100 KB × 1B = 100 TB; index much larger Google, Bing, DuckDuckGo
Case StudyCrawlingDistributed

A web crawler discovers and downloads web pages so a search engine or archive can index them. Google's crawler, Heritrix, Common Crawl, Bing's bot share the architecture. Interesting problems: politeness, scale, crawl traps, freshness.

Scale1B pages, weekly refresh → ~1700 pages/s sustained
StorageCompressed HTML ~100 KB × 1B = 100 TB; index much larger

Key Concepts

1
1. The frontier (queue of URLs to crawl). Must prioritize (PageRank, freshness needs), enforce politeness (one URL per domain at a time, robots.txt crawl delay), and persist (billions of URLs don't fit in memory). Production: partitioned by hash(domain), each partition is a priority queue backed by RocksDB or Kafka. Dedup against a Bloom filter first, then definitive check.
1. The frontier (queue of URLs to crawl).hash(domain)
2
2. Fetchers do the I/O. Pull URLs from frontier → resolve DNS (with cache — DNS is surprisingly expensive at scale) → check robots.txt (cached per domain) → respect crawl delay → fetch HTTPS → follow redirects to a limit → pass content to parser. Async I/O (Go, async Python, Java NIO) is essential — sync would need millions of threads.
2. Fetchers do the I/O.
3
3. Parsing and dedup. Parser extracts outlinks and content. New URLs normalized (lowercase host, sort query params, strip fragment) before adding to frontier — without normalization, the frontier explodes from URL aliases. Dedup by URL canonicalization + SimHash on content for near-duplicate detection (mirrors, paraphrases).
3. Parsing and dedup.
4
4. JS-rendered pages and crawl traps. SPAs return mostly empty HTML — fall back to headless Chromium (Puppeteer, Playwright) at 100x the cost. Detect via empty-content heuristic; pay the cost only when needed. Crawl traps: calendar URLs (?date=YYYY-MM-DD for every day forever), infinite redirect chains. Detect via URL pattern heuristics, throttle or block.
4. JS-rendered pages and crawl traps.?date=YYYY-MM-DD
5
5. Re-crawl scheduling. Track each page's change rate via repeated crawls or sitemap last-mod. Popular news at hourly intervals; static reference pages monthly. Etags and If-Modified-Since save bandwidth on unchanged pages. References: Googlebot (BFS + PageRank + freshness), Heritrix (Internet Archive), Common Crawl (monthly 3B pages, public S3 dataset), Apache Nutch.
5. Re-crawl scheduling.

Related Problems

Design a URL Shortener (TinyURL / bit.ly)mediumDesign Twitter / News FeedhardDesign WhatsApp / Chat SystemhardDesign Uber / Ride-HailinghardDesign Netflix / Video Streaminghard

High-level design

Seed → frontier (partitioned by domain, prioritized).
Fetcher pool → DNS cache → robots.txt cache → HTTP fetch → store HTML.
Parser → outlink extraction → normalization → dedup check → enqueue.
Parser → content extraction → indexing pipeline (separate system).
Recrawl scheduler → re-enqueue pages by change-rate model.
Optional headless fallback for JS-rendered pages.

Components

  • Frontier (Kafka with domain-partitioned queues, or sharded priority queue in RocksDB).
  • Fetcher pool (async HTTP).
  • DNS resolver + cache.
  • Robots.txt cache (TTL per domain).
  • Parser (Beautiful Soup / lxml / native).
  • Canonical URL builder.
  • Dedup: Bloom filter + SimHash for near-dup.
  • Storage: HTML compressed in object store, metadata in KV.
  • Link graph store (for PageRank, recrawl decisions).
  • Recrawl scheduler.
  • Headless browser pool (Puppeteer/Playwright) for SPA fallback.

Politeness

Per-domain delay: robots.txt Crawl-delay directive, or default 1-10s.

One concurrent connection per domain (or low N).

Respect 429 / 503 responses with exponential backoff.

Honor robots.txt strictly — both for ethics and to avoid being banned.

User-Agent identifies your crawler with contact info.

Some sites whitelist specific bots (Googlebot, Bingbot) and may serve different content.

Dedup

URL normalization: lowercase host, sort query params, strip fragment, canonical form.

Bloom filter for 'have we seen this URL?'.

SimHash on content (~64-bit fingerprint of token n-grams) for near-duplicate detection.

MinHash + LSH for cluster-level near-dup at billion scale.

Mirror detection: same content at many hosts; pick canonical version.

Trade-offs

Politeness limits throughput — accept and parallelize across domains.

Frontier in memory: fast but bounded; disk-backed (RocksDB) for scale.

Dedup early (Bloom filter) vs late (content SimHash) — both, in sequence.

Headless browsing: 100x cost, necessary for SPAs.

Aggressive recrawl: fresher index, more bandwidth.

Sparse recrawl: cheap, stale results.

Real-world references

  • Googlebot: undisclosed details; famously polite, BFS+PageRank+freshness-weighted.
  • Heritrix (Internet Archive): open-source, designed for archival crawls.
  • Common Crawl: monthly snapshots of the web, ~3B pages per crawl, public S3 data.
  • Apache Nutch: open-source web crawler with PageRank.
  • Scrapy: not for billion-scale, great for targeted crawls.