Design a Web Crawler
Politely fetch billions of pages, avoid traps and duplicates, and keep a fresh index.
Overview
A web crawler discovers and downloads web pages to build a search index. It's a study in massive-scale BFS: URL frontiers, politeness, deduplication, and freshness, all while being robust to hostile and infinite content.
Requirements
Functional
- Start from seed URLs and follow links to discover the web.
- Download and store page content for indexing.
- Respect robots.txt and per-domain politeness.
- Re-crawl to keep content fresh; avoid re-fetching unchanged pages.
Non-functional
- Massive scale (billions of pages) and horizontally scalable.
- Polite: never overwhelm a single host.
- Robust to crawler traps, duplicate content, and malformed pages.
Back-of-the-envelope
The numbers that justify the architecture.
| Pages to crawl | 10B+ | |
| Crawl rate | ~100K pages/sec | to cover the web monthly |
| Avg page size | ~100 KB | |
| Content storage | ~PB | compressed HTML |
Reference architecture
A URL frontier feeds fetcher workers that download pages, extract links, dedup content, and enqueue new URLs — a distributed BFS with politeness built into the frontier.
Deep dives
The URL frontier
The frontier is the crawler's brain — a set of queues that decide what to fetch next and when. It balances priority (important/fresh pages first) and politeness (never hammer one host). A common design: front queues by priority, back queues by host, with a mapping that guarantees only one worker fetches a given host at a time and enforces a per-host delay. This politeness-by-construction is what keeps you from being banned.
Deduplication at two levels
URL dedup: before enqueuing, check a 'seen' set. At billions of URLs, an exact set is huge — a Bloom filter gives O(1) membership with tiny memory (accepting a small false-positive rate that just means occasionally skipping a new URL). Content dedup: many URLs return identical or near-identical pages. Hash content (checksum for exact; SimHash/shingling for near-duplicates) so you don't index the same content twice.
Politeness and robots
Fetch and cache each domain's `robots.txt`; honor its disallow rules and crawl-delay. Rate-limit per host, identify with a proper User-Agent, and back off on 429/503. DNS resolution is a surprising bottleneck at scale — cache it aggressively.
Traps, freshness, and robustness
Guard against crawler traps (infinite calendars, session-id URLs generating unbounded links) with depth limits and URL pattern heuristics. For freshness, re-crawl on a schedule weighted by how often a page changes (news hourly, archives rarely); use HTTP conditional requests (ETag / If-Modified-Since) to skip unchanged pages cheaply.
Key trade-offs
For each decision: the two options, and when to pick which.
Bottlenecks & follow-ups
Where it breaks under load — and what an interviewer will probe.
- ▲Getting banned → politeness queues + per-host rate limits.
- ▲Memory for the seen set → Bloom filter.
- ▲Crawler traps → depth caps + pattern detection.
- ▲DNS lookups → aggressive DNS caching.
What a strong answer sounds like
- ✓Center the answer on the URL frontier — priority + politeness is the heart of it.
- ✓Bring up Bloom filters for URL dedup; it's the expected scale trick.
- ✓Mention robots.txt and traps unprompted — it signals real-world awareness.