System design template

AI agent orchestration architecture.

The orchestrator owns the loop, and almost every hard problem here is about knowing when to stop.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
AI agent orchestration architecture. 13 components across 6 tiers.
Hover any component to see what it is responsible for.
AI agent orchestration architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
UserClientSupporting component
GatewayEdgeSupporting component
OrchestratorApplicationOwns the loop and decides when a turn ends
Model routerApplicationCheap model plans, strong model executes
Tool registryApplicationWhat the agent may reach, per role
Sub-agentsApplicationRun in isolated worktrees, diff applied back
VerifierApplicationSeparate from the generator, because agents praise their own work
Context managerApplicationCompaction with a retry path on provider overflow
Memory storeDataSupporting component
PostgresDataSupporting component
Task queueDataSupporting component
Model providersExternalSupporting component
TracingInfrastructureSupporting component

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

The verifier must not be the generator

Asking the agent that produced the work whether the work is good returns praise almost regardless of quality, which is documented behaviour rather than a suspicion. Separating the verifier means the judging is done by something with no stake in the answer. It costs another model call per iteration, and it is the difference between a loop that converges and one that declares success immediately.

Route by role, not by preference

Planning, editing and reading images have different requirements, and using one strong model for all of them is expensive for the easy parts. Routing a cheap model to planning and a strong one to the diff is a large cost reduction. The failure is escalating too readily, which gives you the cost of the large model plus the latency of having tried the small one first.

Isolate sub-agent work so failure is free

If a sub-agent writes directly into the working tree, every speculative attempt has a cleanup cost and you allow fewer of them. Running each in an isolated environment and applying the diff back on success makes abandoning an attempt cost nothing, which changes how willing you are to let the system try things.

The loop needs a bound that is not the model's opinion

An agent asked to work until done will sometimes work forever, and sometimes stop immediately. Iteration caps, time limits and token budgets are crude and they are what turn an unbounded process into a bounded one. Every component that enforces one encodes an assumption about what the model cannot do alone, and those assumptions go stale as models improve, so they are worth revisiting rather than treating as permanent.

Tracing is not optional here

When an agent produces the wrong result, the question is whether the plan was wrong, a tool returned something unexpected, or the model ignored what it was given. Those have different fixes and are indistinguishable from the output. Capturing every tool call, its result and the resulting prompt is the difference between improving the system and changing it at random.

How it changes with scale

Cost scales with tokens rather than requests, and the two diverge sharply because a single hard task can consume orders of magnitude more than an easy one. Concurrency is bounded by provider rate limits rather than by your own capacity. The queue is what keeps a burst from turning into unbounded latency, and admission control matters more than throughput.

Where it breaks first

A loop that will not terminate. The verifier rejects, the generator revises, the verifier rejects again, and without an iteration cap this continues until the budget is gone with nothing shipped. The symptom is cost rather than an error, which is why it is usually noticed at the end of the month.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram an AI agent orchestration system: gateway, orchestrator owning the loop, model router, tool registry, sub-agents in isolated environments, a separate verifier, context management, memory store, task queue and tracing.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Why separate the orchestrator from the agent?

Because the orchestrator owns decisions the agent should not make about itself: when to stop, which model to use, what may run without approval, and whether the work passed. An agent that decides its own termination condition tends to decide it has finished.

How many sub-agents should run concurrently?

Few enough that provider rate limits are not the binding constraint and that a human can still review what comes back. Parallelism that produces more output than anyone reads is not throughput.

Should sub-agents be able to spawn their own sub-agents?

Generally no. Unbounded recursion makes cost and duration unpredictable, and the second level rarely improves results enough to justify it.

What does the memory store hold?

Facts worth carrying between sessions: conventions, decisions and summaries of prior work. Not the full transcript, which is what compaction is for and is rarely worth re-reading.

More templates