Webhook delivery architecture.
You are making requests to servers you do not control, which are frequently slow, sometimes wrong, and occasionally gone.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Producing services | Client | Supporting component |
| Subscription API | Application | Endpoints, secrets and event filters |
| Dispatcher | Application | Fans one event to every matching subscription |
| Signer | Application | HMAC with a timestamp, so replays are detectable |
| Delivery workers | Application | Per-endpoint concurrency, so one slow customer is contained |
| Circuit breaker | Application | Pauses an endpoint that keeps failing |
| Retry scheduler | Application | Exponential backoff with jitter |
| Kafka | Data | Supporting component |
| Dead letter queue | Data | Supporting component |
| Postgres | Data | Attempt history, replayable by the customer |
| Customer endpoints | External | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Per-endpoint concurrency, or one customer stalls everyone
A shared worker pool means one customer whose endpoint takes thirty seconds occupies workers that everyone else needs, and during an incident on their side you have an incident on yours. Limiting concurrency per endpoint contains it. The cost is more scheduling state and some idle capacity, which is much cheaper than the coupling it removes.
Sign with a timestamp, not just a body hash
An HMAC over the payload proves it came from you and does nothing against replay: a captured request can be sent again indefinitely. Including a timestamp in the signed material and requiring recipients to reject old ones closes that. It means recipients need roughly correct clocks, which is a support burden you inherit.
Circuit breaking is politeness and self-protection
An endpoint returning errors for an hour will keep doing so, and continuing to retry wastes your workers and hammers someone whose system is already unwell. Pausing that subscription after a threshold and probing occasionally is better for both sides. The risk is pausing on a transient failure, so the threshold has to be forgiving enough not to fire on a blip.
At-least-once, and say so loudly
Guaranteeing exactly-once delivery to an endpoint you do not control is not achievable: a timeout is indistinguishable from a slow success. So delivery is at-least-once, every payload carries a stable event id, and the documentation tells consumers to deduplicate. Pretending otherwise pushes an unsolvable problem onto customers without warning them.
The attempt log is the support tool
Most webhook support tickets are a customer asking whether you sent something. Storing every attempt with its response code and body, visible to the customer, and letting them replay from it, resolves that class of ticket without a human. It costs real storage on high-volume accounts, and it is worth it.
How it changes with scale
Volume is events times matching subscriptions, so a popular event type with many subscribers is the multiplier that surprises people. Throughput is bounded by the slowest endpoints rather than by your own capacity, which makes per-endpoint isolation the thing that determines whether the system degrades gracefully.
Where it breaks first
A slow endpoint rather than a failing one. A dead endpoint fails fast and the breaker opens; one that takes twenty-nine seconds against a thirty-second timeout consumes a worker for the entire window while looking healthy. Without per-endpoint limits and an aggressive timeout, a handful of slow consumers can consume the whole pool while every dashboard says everything is fine.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
More templates
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Chat Application System DesignLong-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.