System design template

LLM inference service architecture.

GPUs are the budget, so almost every decision here is about keeping them busy without letting the queue destroy tail latency.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
LLM inference service architecture. 11 components across 5 tiers.
LLM inference service architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
ClientClientStreams tokens over SSE
API gatewayEdgeSupporting component
Auth & quotaApplicationTokens, not requests, are the unit that matters
Model routerApplicationCheap model first, escalate on difficulty
Request queueApplicationAdmission control protects tail latency
Continuous batcherApplicationBatches at the token level, not per request
GPU workersApplicationSupporting component
Prompt cacheDataPrefix cache, the cheapest speedup available
KV cacheDataLives in GPU memory, sized with batch
Model weightsDataSupporting component
PrometheusInfrastructureWatch queue depth and tokens per second

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

Continuous batching, not request batching

Waiting to collect a batch of requests adds latency to the ones that arrived first, and a batch runs only as fast as its longest generation. Batching at the token level, admitting new requests into the running batch as others finish, keeps utilisation high without that penalty. It is significantly more complex to implement and is the single largest throughput difference between a naive server and a serious one.

The KV cache is the real capacity limit

Memory per request grows with context length, so maximum batch size depends on how long the conversations are, not just how many there are. A service sized against short prompts falls over when users paste in long documents, and the failure is an out-of-memory rather than a graceful slowdown. Capacity planning here has to be in tokens, not requests, which is an uncomfortable change for anyone used to sizing web services.

Prefix caching is the cheapest win available

Shared system prompts and long conversation histories mean the same prefix is processed repeatedly. Caching the computed attention state for that prefix removes most of the prefill work for a large share of traffic. It costs memory that competes with batch size, and it only helps when prompts genuinely share prefixes, which is a property of your product rather than of the server.

Admission control, because a queue is not free capacity

When demand exceeds throughput, an unbounded queue converts an overload into unbounded latency: requests are eventually served, long after the user gave up, having consumed a GPU to produce something nobody reads. Rejecting early with a clear error keeps latency bounded for the requests you do accept. It means visibly failing some traffic under load, which is a decision people find harder to make than it should be.

How it changes with scale

Throughput is tokens per second, not requests per second, and the two diverge sharply because output length varies by orders of magnitude across requests. Scaling out means more GPUs, which is expensive and slow to provision, so routing cheap requests to smaller models is usually a bigger lever than adding capacity.

Where it breaks first

Long-context requests arriving together. Each consumes a large share of KV cache, batch size collapses, throughput falls, the queue grows, and latency degrades for everyone including the short requests that would otherwise have been fast. Without per-request context limits, a handful of users can degrade the service for all of them.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram an LLM inference service: gateway with auth and token quotas, a model router that escalates from cheap to strong models, a request queue with admission control, a continuous batcher, GPU workers with KV cache, a prompt prefix cache, and metrics.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Should you self-host or use an API?

An API until you have a specific reason not to: predictable per-token cost, no GPU procurement, and someone else handles the operational work above. Self-hosting wins on data residency, on sustained high volume where the cost curve crosses, and on models you cannot get otherwise.

How do you route between models?

Cheapest model that can do the job, escalating on signals such as input length, task type or a confidence check. The failure mode is escalating too readily, which gives you the cost of the large model plus the latency of having tried the small one first.

What should you monitor?

Queue depth, time to first token and tokens per second, in that order. Average request latency conflates a short answer with a long one and hides both problems.

Does streaming change the architecture?

It changes the connection handling rather than the inference path: responses are long-lived, so the gateway holds many concurrent open connections. It also changes the metric that matters, because time to first token is what a user perceives as speed, not total duration.

More templates