Web crawler
Download billions of web pages politely and fresh without getting trapped in infinite URL spaces or hammering small hosts. The architecture revolves around a two-tier frontier (priority + per-host politeness queues), URL canonicalisation with Bloom-filter deduplication, SimHash content deduplication, and aggressive DNS/robots.txt caching.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Frontier scheduler 1, Frontier scheduler 2
- Frontier scheduler 1 (service) · calls: URL seen filter (Bloom + KV), robots.txt + DNS cache, Politeness frontier queues, Crawl metadata + hashes
- Frontier scheduler 2 (service) · calls: URL seen filter (Bloom + KV), robots.txt + DNS cache, Politeness frontier queues, Crawl metadata + hashes
- URL seen filter (Bloom + KV) (dedupStore) · calls: none
- robots.txt + DNS cache (cache) · calls: none
- Politeness frontier queues (queue) · calls: none
- Fetcher + parser workers (worker) · calls: robots.txt + DNS cache, Raw HTML archive, Crawl metadata + hashes
- Crawl metadata + hashes (database) · calls: none
- Raw HTML archive (objectStore) · calls: none
Requirements
Functional
- Start from seed URLs, fetch pages, parse outgoing links, and enqueue unseen URLs.
- Enforce politeness: obey robots.txt rules, Crawl-delay, and a maximum of one concurrent request per host.
- Deduplicate both URLs (after canonicalisation) and page contents (near-duplicate detection).
- Prioritise high-value and frequently updated pages for recrawling.
Non-functional
- Crawl ~15 billion pages per month (~6,000 pages/s sustained, ~12,000/s peak).
- Never overload a target web server; strict per-host rate limiting even across hundreds of fetcher nodes.
- Resilient to spider traps, slow responses, malformed HTML, and crashes without losing frontier state.
Worked estimates
Crawl rate
≈ 5,800 pages/s average, ≈ 12,000 pages/s peak
15B pages/month ÷ (30 × 86,400 s)
Ingress bandwidth
≈ 580 MB/s (4.6 Gbit/s) sustained; peak ~10 Gbit/s
5,800 pages/s × ~100 KB average compressed HTML
HTML storage
≈ 1.5 PB per crawl cycle; multi-petabyte object store over retention
15B pages × 100 KB compressed
URL seen filter (Bloom)
≈ 26 GB of RAM — easily fits in memory across a small cluster
15B URLs × ~14 bits/URL (≈ 0.1% false-positive rate)
Fetcher concurrency
≈ 3,000 concurrent fetches in flight; async I/O handles thousands per worker
Little's law: 6,000 pages/s × 500 ms mean fetch time
API
POST /v1/frontier/urls
Request: { urls: [{ rawUrl, sourceHost, depth }] }
Response: 201 { enqueued, skippedDuplicate }
Internal RPC from parser workers feeding discovered links back to the frontier.
GET /v1/robots/{domain}
Request: —
Response: 200 { disallow[], crawlDelayMs, expiresAt }
Cached per host; fetched at most once per TTL.
Data model
- pages — store: pagedb
- htmlBlobs — store: blobstore
- robotsRules — store: pagedb
urlseen: an in-memory sharded Bloom filter backed by a persistent KV table of canonicalised URL hashes so a restart does not recrawl the web.
robotsRules: cached per registered host for up to 24 hours so fetchers do not re-request /robots.txt on every page.
pages: stores metadata, HTTP status, fetch timestamp, and a 64-bit SimHash fingerprint to detect near-duplicate content before indexing.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Frontier architecture: priority + politeness
Following Mercator's classic design, separate the frontier into two stages. Front queues rank URLs by priority (PageRank, domain authority, update frequency, freshness). A router moves URLs from front queues into back queues — one FIFO queue per target host — and a heap tracks the next allowed fetch time `t_next = t_last + crawlDelay` for each back queue. Assigning each host consistently to one fetcher worker ensures two workers never hit the same host concurrently.
URL canonicalisation, Bloom filters, and spider traps
Before checking whether a URL was seen, normalise it: lowercase scheme and host, strip default ports (`:80`, `:443`) and fragments, sort query parameters, resolve relative paths, and strip known tracking parameters (`utm_*`). Check the canonical URL against a sharded Bloom filter. To defeat spider traps (infinite calendars, session IDs in paths, faceted search combinations), enforce a maximum URL length, maximum path depth, and a per-host page budget.
Content deduplication with SimHash
Up to 30% of web pages are exact or near-duplicates (mirrors, print views, syndication, boilerplate changes). Cryptographic hashes (SHA-256) only catch byte-identical pages. Charikar's SimHash hashes token features into a 64-bit fingerprint such that two pages with similar text have a small Hamming distance (e.g. ≤ 3 bits differing), catching near-duplicates before wasting index storage.
Failure scenarios
Link discovery burst from hub pages
Scenario and expected result
From t = 1.2 s to 2.1 s crawlers hit large directory/sitemap pages and URL enqueue traffic doubles.
Expected: The Bloom/dedup filter absorbs duplicate URLs in sub-millisecond time, and the partitioned frontier queue buffers the newly discovered URLs so fetcher workers drain them at a steady, polite pace.
Fetcher worker pool crashes mid-crawl
Scenario and expected result
Fetcher workers crash from t = 1.3 s to 1.8 s while holding leased URLs from the frontier queue.
Expected: Frontier schedulers keep accepting discovered URLs and serving robots.txt lookups. Unacknowledged URLs in the frontier queue reappear after the visibility timeout and are fetched once workers recover; no frontier state is lost.
Object store write latency spikes 4×
Scenario and expected result
From t = 1.4 s to 2.2 s writing raw HTML blobs to the object store takes 4× longer.
Expected: Because HTML writes happen asynchronously inside the fetcher workers behind the frontier queue, URL ingestion and robots.txt reads stay fast while the frontier queue temporarily buffers the slower fetch completion rate.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No seen-URL or robots.txt cache: hit metadata DB on every URL
- Every discovered URL and page fetch checks URL seen state and robots.txt rules. Without in-memory Bloom/robots caches, every request scans the metadata database and saturates it immediately.
Reference
- Inspect the architecture above and run the same scenario.
Single FIFO queue instead of per-host politeness queues
- A single FIFO frontier is trivial to build, but a page with 500 internal links enqueues 500 consecutive requests to the same host, turning your crawler into a denial-of-service attacker. Splitting the frontier into a front priority tier and a back tier of per-host FIFO queues mapped to specific fetcher workers guarantees politeness.
Reference
- Inspect the architecture above and run the same scenario.
Exact hash set in database instead of a Bloom filter
- Every parsed page yields ~50–100 outgoing URLs, so URL deduplication runs at 50–100× the page fetch rate (~500k checks/s). Doing 500k disk lookups/s is expensive; an in-memory Bloom filter filters out 99.9% of already-seen URLs in RAM at ~14 bits per URL.
Clarifying questions
- Are we indexing HTML text only, or also rendering JavaScript (headless browsers) and downloading media?
- How many pages per month, and do we recrawl continuously for freshness or run one-off snapshots?
- What is our tolerance for Bloom filter false positives (skipping a tiny fraction of unseen URLs)?
- How should we handle paywalls, authentication, and geo-specific content?
Common mistakes
- Using a single FIFO queue, which hammers a single host with bursts of requests.
- Using synchronous DNS resolution and fetching robots.txt on every page.
- Checking URL deduplication without canonicalising URLs first.
- Using MD5/SHA-256 alone for content deduplication, missing near-duplicate pages.
- Ignoring spider traps like infinite calendar links or session IDs embedded in URLs.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Fetch pages from seeds.
- 2/4 Mentions robots.txt or deduplication.
- 3/4 Politeness, URL + content dedup, freshness, and scale targets.
- 4/4 Defines spider-trap defences, recrawl priority, and failure recovery.
estimates / levels 1-4
- 1/4 None.
- 2/4 Pages per second.
- 3/4 Pages/s, bandwidth, HTML storage, and URL dedup memory.
- 4/4 Sizes Bloom filter bits per URL and fetcher concurrency via Little's law.
api / levels 1-4
- 1/4 Seed endpoint.
- 2/4 Enqueue and fetch interfaces.
- 3/4 Batch URL submission, robots.txt cache API, and status reporting.
- 4/4 Idempotent frontier leases and canonicalisation rules.
data model / levels 1-4
- 1/4 URLs table.
- 2/4 URLs and pages.
- 3/4 Frontier state, seen-URL filter, robots cache, and SimHash fingerprints.
- 4/4 Separates hot metadata/fingerprints from cold compressed HTML blobs.
high level / levels 1-4
- 1/4 Workers fetching URLs from a list.
- 2/4 Queue + fetchers + storage.
- 3/4 Two-tier frontier, DNS/robots caches, parser feeding URL dedup.
- 4/4 Consistent host-to-worker mapping and back-pressure when stores slow down.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions Bloom filters.
- 3/4 Mercator front/back queues and Bloom filter false-positive trade-offs.
- 4/4 SimHash near-duplicate detection and spider-trap heuristics.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Crawl rate, host errors, and frontier lag as key signals.
- 4/4 SLOs on politeness compliance, freshness, and trap detection alerts.