Distributed web crawler architecture.
The engineering problem is not fetching pages quickly. It is fetching them politely, at scale, without visiting the same content a thousand times.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Seed list | Client | Supporting component |
| URL frontier | Application | Priority plus politeness delay per host |
| Fetcher pool | Application | Respects robots.txt and crawl delay |
| Robots cache | Application | Supporting component |
| Parser | Application | Supporting component |
| Dedup | Application | Content hash, not URL: mirrors are everywhere |
| Indexer | Application | Supporting component |
| Kafka | Data | Supporting component |
| Seen set | Data | Bloom filter, false positives are acceptable |
| Raw pages | Data | Supporting component |
| Search index | Data | Supporting component |
| The web | External | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Politeness is a per-host constraint on a global queue
A naive priority queue will happily send a thousand simultaneous requests to one host, which is indistinguishable from an attack. The frontier therefore has to be structured per host, with its own delay, while still supporting global prioritisation. That dual requirement is the reason the frontier is a component rather than a data structure, and it means the crawler is often idle on high-priority work because the relevant host is in a cooldown.
Deduplicate on content, not on URL
The same page is reachable through many URLs: tracking parameters, session ids, mirrors, print views. URL-based deduplication catches almost none of it. Hashing normalised content catches all of it, at the cost of fetching the page before you can tell, so you pay the bandwidth either way and save only the parsing and indexing.
A Bloom filter, accepting that it lies occasionally
Tracking every URL ever seen exactly requires memory proportional to the web. A Bloom filter answers in constant space with false positives but never false negatives, meaning the crawler occasionally skips a page it has not actually seen. For a crawler that is an acceptable loss, and choosing it is a deliberate decision to trade completeness for tractability rather than an optimisation.
Store raw pages before parsing them
Parsers change constantly, and reparsing from stored HTML is far cheaper and far politer than recrawling. Keeping raw content makes the extraction logic iterable. It costs a great deal of storage, which is the main reason people skip it and then discover the cost the first time they need to change an extractor.
How it changes with scale
Throughput is bounded by politeness rather than bandwidth: you can only go so fast per host, so crawling faster means crawling more hosts concurrently. That makes frontier partitioning by host the natural sharding key, and it makes the long tail of small sites, rather than the large ones, the thing that determines total coverage.
Where it breaks first
A crawler trap. Dynamically generated pages that link to more dynamically generated pages, calendars being the classic case, produce an infinite supply of unique URLs with no new content. Without depth limits, per-host budgets and content-based deduplication, the crawler spends itself entirely on one site and never notices.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
How strictly should robots.txt be followed?
Strictly, and cache it with a sensible TTL rather than fetching it per request. Ignoring it gets you blocked, and at any scale it gets you blocked by the hosts you most wanted.
How do you decide what to crawl next?
Some combination of expected value and freshness: pages that change often and matter often, recrawled sooner. This is where crawlers actually differ, and it is much more consequential than fetch speed.
Should you render JavaScript?
It costs one to two orders of magnitude more per page than fetching HTML, so it is a per-site decision rather than a default. Many sites still expose the content in the initial response or a feed if you look.
How do you avoid recrawling unchanged pages?
Conditional requests with ETag and If-Modified-Since. A 304 costs a round trip and no bandwidth, and well-behaved sites support it, which makes it the cheapest freshness mechanism available.
More templates
- Search Autocomplete System DesignThis is a latency problem wearing a search problem's clothes: past about a hundred milliseconds users have already typed the next character and your response is worthless.
- File Storage Service DesignThe design goal is to keep file bytes away from your servers entirely, so that what you run is a metadata service that happens to be about files.
- Ticket Booking System DesignA named seat can be sold exactly once, and ten thousand people want it at the same instant. Everything here follows from that.
- Food Delivery System DesignThree sides to coordinate and one deadline that nobody controls: the kitchen. Dispatch quality matters more here than any amount of throughput.
Last updated