Distributed web crawler architecture.
The engineering problem is not fetching pages quickly. It is fetching them politely, at scale, without visiting the same content a thousand times.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Seed list | Client | Supporting component |
| URL frontier | Application | Priority plus politeness delay per host |
| Fetcher pool | Application | Respects robots.txt and crawl delay |
| Robots cache | Application | Supporting component |
| Parser | Application | Supporting component |
| Dedup | Application | Content hash, not URL: mirrors are everywhere |
| Indexer | Application | Supporting component |
| Kafka | Data | Supporting component |
| Seen set | Data | Bloom filter, false positives are acceptable |
| Raw pages | Data | Supporting component |
| Search index | Data | Supporting component |
| The web | External | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Politeness is a per-host constraint on a global queue
A naive priority queue will happily send a thousand simultaneous requests to one host, which is indistinguishable from an attack. The frontier therefore has to be structured per host, with its own delay, while still supporting global prioritisation. That dual requirement is the reason the frontier is a component rather than a data structure, and it means the crawler is often idle on high-priority work because the relevant host is in a cooldown.
Deduplicate on content, not on URL
The same page is reachable through many URLs: tracking parameters, session ids, mirrors, print views. URL-based deduplication catches almost none of it. Hashing normalised content catches all of it, at the cost of fetching the page before you can tell, so you pay the bandwidth either way and save only the parsing and indexing.
A Bloom filter, accepting that it lies occasionally
Tracking every URL ever seen exactly requires memory proportional to the web. A Bloom filter answers in constant space with false positives but never false negatives, meaning the crawler occasionally skips a page it has not actually seen. For a crawler that is an acceptable loss, and choosing it is a deliberate decision to trade completeness for tractability rather than an optimisation.
Store raw pages before parsing them
Parsers change constantly, and reparsing from stored HTML is far cheaper and far politer than recrawling. Keeping raw content makes the extraction logic iterable. It costs a great deal of storage, which is the main reason people skip it and then discover the cost the first time they need to change an extractor.
How it changes with scale
Throughput is bounded by politeness rather than bandwidth: you can only go so fast per host, so crawling faster means crawling more hosts concurrently. That makes frontier partitioning by host the natural sharding key, and it makes the long tail of small sites, rather than the large ones, the thing that determines total coverage.
Where it breaks first
A crawler trap. Dynamically generated pages that link to more dynamically generated pages, calendars being the classic case, produce an infinite supply of unique URLs with no new content. Without depth limits, per-host budgets and content-based deduplication, the crawler spends itself entirely on one site and never notices.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
More templates
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Chat Application System DesignLong-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.