LLM inference service architecture.
GPUs are the budget, so almost every decision here is about keeping them busy without letting the queue destroy tail latency.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Client | Client | Streams tokens over SSE |
| API gateway | Edge | Supporting component |
| Auth & quota | Application | Tokens, not requests, are the unit that matters |
| Model router | Application | Cheap model first, escalate on difficulty |
| Request queue | Application | Admission control protects tail latency |
| Continuous batcher | Application | Batches at the token level, not per request |
| GPU workers | Application | Supporting component |
| Prompt cache | Data | Prefix cache, the cheapest speedup available |
| KV cache | Data | Lives in GPU memory, sized with batch |
| Model weights | Data | Supporting component |
| Prometheus | Infrastructure | Watch queue depth and tokens per second |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Continuous batching, not request batching
Waiting to collect a batch of requests adds latency to the ones that arrived first, and a batch runs only as fast as its longest generation. Batching at the token level, admitting new requests into the running batch as others finish, keeps utilisation high without that penalty. It is significantly more complex to implement and is the single largest throughput difference between a naive server and a serious one.
The KV cache is the real capacity limit
Memory per request grows with context length, so maximum batch size depends on how long the conversations are, not just how many there are. A service sized against short prompts falls over when users paste in long documents, and the failure is an out-of-memory rather than a graceful slowdown. Capacity planning here has to be in tokens, not requests, which is an uncomfortable change for anyone used to sizing web services.
Prefix caching is the cheapest win available
Shared system prompts and long conversation histories mean the same prefix is processed repeatedly. Caching the computed attention state for that prefix removes most of the prefill work for a large share of traffic. It costs memory that competes with batch size, and it only helps when prompts genuinely share prefixes, which is a property of your product rather than of the server.
Admission control, because a queue is not free capacity
When demand exceeds throughput, an unbounded queue converts an overload into unbounded latency: requests are eventually served, long after the user gave up, having consumed a GPU to produce something nobody reads. Rejecting early with a clear error keeps latency bounded for the requests you do accept. It means visibly failing some traffic under load, which is a decision people find harder to make than it should be.
How it changes with scale
Throughput is tokens per second, not requests per second, and the two diverge sharply because output length varies by orders of magnitude across requests. Scaling out means more GPUs, which is expensive and slow to provision, so routing cheap requests to smaller models is usually a bigger lever than adding capacity.
Where it breaks first
Long-context requests arriving together. Each consumes a large share of KV cache, batch size collapses, throughput falls, the queue grows, and latency degrades for everyone including the short requests that would otherwise have been fast. Without per-request context limits, a handful of users can degrade the service for all of them.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
Should you self-host or use an API?
An API until you have a specific reason not to: predictable per-token cost, no GPU procurement, and someone else handles the operational work above. Self-hosting wins on data residency, on sustained high volume where the cost curve crosses, and on models you cannot get otherwise.
How do you route between models?
Cheapest model that can do the job, escalating on signals such as input length, task type or a confidence check. The failure mode is escalating too readily, which gives you the cost of the large model plus the latency of having tried the small one first.
What should you monitor?
Queue depth, time to first token and tokens per second, in that order. Average request latency conflates a short answer with a long one and hides both problems.
Does streaming change the architecture?
It changes the connection handling rather than the inference path: responses are long-lived, so the gateway holds many concurrent open connections. It also changes the metric that matters, because time to first token is what a user perceives as speed, not total duration.
More templates
- CI/CD Pipeline ArchitectureTwo things decide whether people trust a pipeline: how fast it is, and whether a red build actually means something is broken.
- Ride Sharing System DesignEvery driver streams their position every few seconds whether or not anyone is riding, which makes this one of the few consumer systems where writes dominate reads.
- Video Streaming System DesignAround ninety-nine percent of the bytes never touch your servers, which means most of this design is about preparing files so the CDN can do the work.
- Social Media Feed System DesignFan-out on write is the right answer until someone with ten million followers posts, and the entire design exists to handle that exception.
Last updated