Web crawler

Download billions of web pages politely and fresh without getting trapped in infinite URL spaces or hammering small hosts. The architecture revolves around a two-tier frontier (priority + per-host politeness queues), URL canonicalisation with Bloom-filter deduplication, SimHash content deduplication, and aggressive DNS/robots.txt caching.

Notes and ratings save in this browser. Completing self-assessment records your progress.

Duration 45:00

Requirements checklist

Start from seed URLs, fetch pages, parse outgoing links, and enqueue unseen URLs.Enforce politeness: obey robots.txt rules, Crawl-delay, and a maximum of one concurrent request per host.Deduplicate both URLs (after canonicalisation) and page contents (near-duplicate detection).Prioritise high-value and frequently updated pages for recrawling.Crawl ~15 billion pages per month (~6,000 pages/s sustained, ~12,000/s peak).Never overload a target web server; strict per-host rate limiting even across hundreds of fetcher nodes.Resilient to spider traps, slow responses, malformed HTML, and crashes without losing frontier state.

Clarifying questions

Are we indexing HTML text only, or also rendering JavaScript (headless browsers) and downloading media?
How many pages per month, and do we recrawl continuously for freshness or run one-off snapshots?
What is our tolerance for Bloom filter false positives (skipping a tiny fraction of unseen URLs)?
How should we handle paywalls, authentication, and geo-specific content?

Open a practice canvas (new tab)Estimation calculators (new tab)

requirements

Suggested time: about 6 minutes. Defines spider-trap defences, recrawl priority, and failure recovery.

Notes for requirements