System design template

Distributed rate limiter architecture.

Every design here is a trade between how accurate the limit is and how much latency you are willing to add to every single request to achieve it.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
Distributed rate limiter architecture. 11 components across 5 tiers.
Distributed rate limiter architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
ClientClientSupporting component
Edge proxyEdgeFirst rejection point, cheapest place to say no
API gatewayEdgeSupporting component
Limiter middlewareApplicationToken bucket, evaluated per request
Local counterApplicationIn-process, absorbs most checks
Sync workerApplicationReconciles local drift with the shared store
Policy serviceApplicationPer plan and per route quotas
Upstream servicesApplicationSupporting component
RedisDataShared counters, Lua for atomicity
PostgresDataSupporting component
PrometheusInfrastructureRejection rate is the signal to watch

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

Reject as early as possible

The cheapest rejection is the one that never reaches your application. Pushing coarse limits to the edge proxy means abusive traffic costs you a connection rather than a request path, which is the difference between absorbing an attack and being taken down by one. The edge cannot see per-plan quotas or user identity, so this only ever handles the blunt cases and a second layer is still required.

Local counters trade accuracy for latency

Checking a shared store on every request adds a network round trip to your p50. Counting locally and reconciling periodically removes that, at the cost of allowing more through than the limit strictly permits, bounded by the number of nodes times the sync interval. For protecting a system from overload, approximate is entirely fine. For a quota someone is billed against, it is not, and that distinction should decide the design rather than a preference for one algorithm.

Token bucket, because bursts are normal

A fixed window rejects the eleventh request in a second even when the previous nine seconds were idle, which is a poor description of how clients actually behave, and it produces a stampede at every window boundary. A token bucket permits a burst up to the bucket size and then enforces a sustained rate, which matches both real traffic and what users expect. It costs two values per key rather than one and needs atomic updates, which is what the Lua script is for.

Policy separate from enforcement

Hardcoding limits in the middleware means a plan change is a deploy. Loading them from a policy service makes limits data, at the cost of a lookup that must be cached aggressively and a decision about what happens when the policy service is unavailable. Failing open there is usually right, since a limiter that blocks everything because it cannot read its config is a worse outage than briefly unlimited traffic.

How it changes with scale

Cost is per request rather than per user, so it grows with total traffic including the traffic you are rejecting. Key cardinality is the thing that surprises people: limiting per user per route multiplies out quickly, and the shared store ends up holding far more keys than expected. Short TTLs are what keep that bounded.

Where it breaks first

The shared store becoming unavailable. Every request now needs a decision with no shared state, and the choice made in advance determines whether you fail open, accepting unlimited traffic, or fail closed, rejecting everything. Both are bad; the failure mode here is not having decided, so the behaviour is whatever the client library's default timeout does.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram a distributed rate limiter: edge proxy rejection, limiter middleware with a local in-process counter, a shared Redis store with Lua for atomic token bucket updates, a policy service for per-plan quotas, and metrics.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Token bucket or leaky bucket?

Token bucket for APIs, because it permits the bursts real clients produce. Leaky bucket enforces a perfectly smooth output rate, which is what you want when protecting something that genuinely cannot absorb a burst, such as a downstream with fixed concurrency.

What should a rejected request return?

429, with a Retry-After header. Without that header clients retry immediately and often, which is the single largest source of load during a rejection event.

Should rate limit state be replicated across regions?

Usually not. Cross-region consistency costs more latency than the accuracy is worth. Per-region limits sized to the regional share are simpler and fail more gracefully.

How do you rate limit unauthenticated traffic?

By IP, knowing it is a poor identifier: shared NATs mean many users behind one address and attackers rotate addresses cheaply. It is a mitigation rather than a control, which is why unauthenticated limits should be tight and authenticated ones generous.

More templates