Distributed rate limiter architecture.
Every design here is a trade between how accurate the limit is and how much latency you are willing to add to every single request to achieve it.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Client | Client | Supporting component |
| Edge proxy | Edge | First rejection point, cheapest place to say no |
| API gateway | Edge | Supporting component |
| Limiter middleware | Application | Token bucket, evaluated per request |
| Local counter | Application | In-process, absorbs most checks |
| Sync worker | Application | Reconciles local drift with the shared store |
| Policy service | Application | Per plan and per route quotas |
| Upstream services | Application | Supporting component |
| Redis | Data | Shared counters, Lua for atomicity |
| Postgres | Data | Supporting component |
| Prometheus | Infrastructure | Rejection rate is the signal to watch |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Reject as early as possible
The cheapest rejection is the one that never reaches your application. Pushing coarse limits to the edge proxy means abusive traffic costs you a connection rather than a request path, which is the difference between absorbing an attack and being taken down by one. The edge cannot see per-plan quotas or user identity, so this only ever handles the blunt cases and a second layer is still required.
Local counters trade accuracy for latency
Checking a shared store on every request adds a network round trip to your p50. Counting locally and reconciling periodically removes that, at the cost of allowing more through than the limit strictly permits, bounded by the number of nodes times the sync interval. For protecting a system from overload, approximate is entirely fine. For a quota someone is billed against, it is not, and that distinction should decide the design rather than a preference for one algorithm.
Token bucket, because bursts are normal
A fixed window rejects the eleventh request in a second even when the previous nine seconds were idle, which is a poor description of how clients actually behave, and it produces a stampede at every window boundary. A token bucket permits a burst up to the bucket size and then enforces a sustained rate, which matches both real traffic and what users expect. It costs two values per key rather than one and needs atomic updates, which is what the Lua script is for.
Policy separate from enforcement
Hardcoding limits in the middleware means a plan change is a deploy. Loading them from a policy service makes limits data, at the cost of a lookup that must be cached aggressively and a decision about what happens when the policy service is unavailable. Failing open there is usually right, since a limiter that blocks everything because it cannot read its config is a worse outage than briefly unlimited traffic.
How it changes with scale
Cost is per request rather than per user, so it grows with total traffic including the traffic you are rejecting. Key cardinality is the thing that surprises people: limiting per user per route multiplies out quickly, and the shared store ends up holding far more keys than expected. Short TTLs are what keep that bounded.
Where it breaks first
The shared store becoming unavailable. Every request now needs a decision with no shared state, and the choice made in advance determines whether you fail open, accepting unlimited traffic, or fail closed, rejecting everything. Both are bad; the failure mode here is not having decided, so the behaviour is whatever the client library's default timeout does.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
More templates
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Chat Application System DesignLong-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.