Webhook delivery architecture.
You are making requests to servers you do not control, which are frequently slow, sometimes wrong, and occasionally gone.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Producing services | Client | Supporting component |
| Subscription API | Application | Endpoints, secrets and event filters |
| Dispatcher | Application | Fans one event to every matching subscription |
| Signer | Application | HMAC with a timestamp, so replays are detectable |
| Delivery workers | Application | Per-endpoint concurrency, so one slow customer is contained |
| Circuit breaker | Application | Pauses an endpoint that keeps failing |
| Retry scheduler | Application | Exponential backoff with jitter |
| Kafka | Data | Supporting component |
| Dead letter queue | Data | Supporting component |
| Postgres | Data | Attempt history, replayable by the customer |
| Customer endpoints | External | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Per-endpoint concurrency, or one customer stalls everyone
A shared worker pool means one customer whose endpoint takes thirty seconds occupies workers that everyone else needs, and during an incident on their side you have an incident on yours. Limiting concurrency per endpoint contains it. The cost is more scheduling state and some idle capacity, which is much cheaper than the coupling it removes.
Sign with a timestamp, not just a body hash
An HMAC over the payload proves it came from you and does nothing against replay: a captured request can be sent again indefinitely. Including a timestamp in the signed material and requiring recipients to reject old ones closes that. It means recipients need roughly correct clocks, which is a support burden you inherit.
Circuit breaking is politeness and self-protection
An endpoint returning errors for an hour will keep doing so, and continuing to retry wastes your workers and hammers someone whose system is already unwell. Pausing that subscription after a threshold and probing occasionally is better for both sides. The risk is pausing on a transient failure, so the threshold has to be forgiving enough not to fire on a blip.
At-least-once, and say so loudly
Guaranteeing exactly-once delivery to an endpoint you do not control is not achievable: a timeout is indistinguishable from a slow success. So delivery is at-least-once, every payload carries a stable event id, and the documentation tells consumers to deduplicate. Pretending otherwise pushes an unsolvable problem onto customers without warning them.
The attempt log is the support tool
Most webhook support tickets are a customer asking whether you sent something. Storing every attempt with its response code and body, visible to the customer, and letting them replay from it, resolves that class of ticket without a human. It costs real storage on high-volume accounts, and it is worth it.
How it changes with scale
Volume is events times matching subscriptions, so a popular event type with many subscribers is the multiplier that surprises people. Throughput is bounded by the slowest endpoints rather than by your own capacity, which makes per-endpoint isolation the thing that determines whether the system degrades gracefully.
Where it breaks first
A slow endpoint rather than a failing one. A dead endpoint fails fast and the breaker opens; one that takes twenty-nine seconds against a thirty-second timeout consumes a worker for the entire window while looking healthy. Without per-endpoint limits and an aggressive timeout, a handful of slow consumers can consume the whole pool while every dashboard says everything is fine.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
How long should retries continue?
Long enough to survive a deploy on the receiving side, typically hours with exponential backoff and jitter, then to the dead letter queue. Retrying for days delivers events whose usefulness expired long before.
How should the receiver verify a webhook?
Recompute the HMAC over the raw body with the shared secret and compare in constant time, then check the timestamp is recent. Parsing the body before verifying is a common and avoidable mistake.
Should webhooks be ordered?
Avoid promising it. Ordering across retries and concurrent deliveries is expensive and constrains throughput. Send a sequence number and let consumers order if they care.
What response should the receiver return?
A 2xx as soon as the event is durably queued on their side, not after processing it. Receivers that do the work synchronously before responding are the main source of timeouts.
More templates
- Audit Logging System DesignAn audit log that can be edited is not an audit log, which makes this one of the few systems where the absence of features is the design.
- Vector Search Service DesignThe failure mode here is not an error. It is recall dropping slowly as the index grows, with nothing in your metrics saying so.
- AI Agent Orchestration DesignThe orchestrator owns the loop, and almost every hard problem here is about knowing when to stop.
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
Last updated