AI agent orchestration architecture.
The orchestrator owns the loop, and almost every hard problem here is about knowing when to stop.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| User | Client | Supporting component |
| Gateway | Edge | Supporting component |
| Orchestrator | Application | Owns the loop and decides when a turn ends |
| Model router | Application | Cheap model plans, strong model executes |
| Tool registry | Application | What the agent may reach, per role |
| Sub-agents | Application | Run in isolated worktrees, diff applied back |
| Verifier | Application | Separate from the generator, because agents praise their own work |
| Context manager | Application | Compaction with a retry path on provider overflow |
| Memory store | Data | Supporting component |
| Postgres | Data | Supporting component |
| Task queue | Data | Supporting component |
| Model providers | External | Supporting component |
| Tracing | Infrastructure | Supporting component |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
The verifier must not be the generator
Asking the agent that produced the work whether the work is good returns praise almost regardless of quality, which is documented behaviour rather than a suspicion. Separating the verifier means the judging is done by something with no stake in the answer. It costs another model call per iteration, and it is the difference between a loop that converges and one that declares success immediately.
Route by role, not by preference
Planning, editing and reading images have different requirements, and using one strong model for all of them is expensive for the easy parts. Routing a cheap model to planning and a strong one to the diff is a large cost reduction. The failure is escalating too readily, which gives you the cost of the large model plus the latency of having tried the small one first.
Isolate sub-agent work so failure is free
If a sub-agent writes directly into the working tree, every speculative attempt has a cleanup cost and you allow fewer of them. Running each in an isolated environment and applying the diff back on success makes abandoning an attempt cost nothing, which changes how willing you are to let the system try things.
The loop needs a bound that is not the model's opinion
An agent asked to work until done will sometimes work forever, and sometimes stop immediately. Iteration caps, time limits and token budgets are crude and they are what turn an unbounded process into a bounded one. Every component that enforces one encodes an assumption about what the model cannot do alone, and those assumptions go stale as models improve, so they are worth revisiting rather than treating as permanent.
Tracing is not optional here
When an agent produces the wrong result, the question is whether the plan was wrong, a tool returned something unexpected, or the model ignored what it was given. Those have different fixes and are indistinguishable from the output. Capturing every tool call, its result and the resulting prompt is the difference between improving the system and changing it at random.
How it changes with scale
Cost scales with tokens rather than requests, and the two diverge sharply because a single hard task can consume orders of magnitude more than an easy one. Concurrency is bounded by provider rate limits rather than by your own capacity. The queue is what keeps a burst from turning into unbounded latency, and admission control matters more than throughput.
Where it breaks first
A loop that will not terminate. The verifier rejects, the generator revises, the verifier rejects again, and without an iteration cap this continues until the budget is gone with nothing shipped. The symptom is cost rather than an error, which is why it is usually noticed at the end of the month.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
More templates
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Chat Application System DesignLong-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.