System design template

RAG pipeline architecture.

Almost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
RAG pipeline architecture. 12 components across 4 tiers.
RAG pipeline architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
ClientClientSupporting component
Query APIApplicationSupporting component
Query rewriterApplicationExpands the question before retrieval
RetrieverApplicationHybrid: dense vectors plus BM25
RerankerApplicationCross-encoder over the top 50
GeneratorApplicationAnswers only from retrieved context
Ingest workerApplicationChunk, embed, upsert
Embedding modelApplicationSupporting component
QdrantDataSupporting component
Document storeDataSupporting component
PostgresDataSupporting component
TracingInfrastructureRetrieval quality is only visible in traces

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

Hybrid retrieval, because embeddings miss exact terms

Dense vectors are good at meaning and bad at specifics: part numbers, error codes, proper nouns and anything rare in the training distribution. Keyword search is the reverse. Running both and fusing the results costs a second index to maintain and a fusion parameter to tune, and it removes an entire class of failure where a user searches for the exact string in a document and gets nothing.

Reranking is where the quality is

Retrieving fifty candidates cheaply and then scoring them with a cross-encoder that reads query and document together is consistently better than retrieving ten and hoping. The cross-encoder cannot be precomputed, because it needs the pair, so it adds real latency proportional to the candidate count. That tradeoff, a slower answer that is right, is usually the correct one, and it is the single highest-leverage change in most underperforming RAG systems.

Chunking decides the ceiling and cannot be fixed later

Chunks that are too small lose the context that makes them interpretable; chunks that are too large dilute the embedding until it matches everything weakly. Whatever you pick is baked into the index, so changing it means re-embedding the entire corpus. This is the decision most worth prototyping properly before committing, and the one most often made by accepting a default.

Without tracing you are guessing

When an answer is wrong, the question is whether retrieval returned the wrong documents, the reranker ordered them badly, or the generator ignored what it was given. Those have completely different fixes and are indistinguishable from the output alone. Capturing the retrieved set, the scores and the final prompt for every request costs storage and adds a dependency, and it is the difference between improving the system and changing it at random.

How it changes with scale

Query volume drives the retrieval and reranking tiers, and reranking is the expensive one, so it is the first thing to cache or shrink. Corpus size drives the vector store, and vector search degrades gracefully until it does not: recall at a fixed latency drops as the index grows, which is a quality regression that no error rate will show you. Ingest is bursty by nature and belongs on its own workers so a bulk reindex cannot starve live queries.

Where it breaks first

Silent retrieval degradation. Nothing errors, latency looks normal, and answers slowly get worse as the corpus grows or drifts away from the queries people ask. Because there is no exception and no alert, this is usually discovered through user complaints months later, which is the argument for evaluating retrieval on a fixed question set continuously rather than only at launch.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram a RAG pipeline: query API, query rewriting, hybrid retrieval over a vector database, a cross-encoder reranker, a generator, plus an ingest path that chunks, embeds and upserts documents. Include tracing.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Do you need a dedicated vector database?

Not always. Postgres with pgvector is sufficient well past the point most teams assume, and it keeps your documents and their embeddings in one system with one backup story. A dedicated store earns its place when the index no longer fits comfortably in memory or when you need filtering and vector search to be fast at the same time.

How large should chunks be?

Large enough to stand alone as an answer to something, small enough that the embedding still means one thing. Splitting on document structure, headings and sections, beats splitting on a fixed token count, because the structure usually encodes exactly the boundary you want.

Can this run entirely locally?

Yes. Local embedding and generation models with a local vector store is a complete offline pipeline, which matters when the corpus is the sort of thing you cannot send to a third party. Quality is lower than frontier hosted models, though the gap on retrieval-grounded answering is much smaller than on open-ended generation.

How do you stop it inventing answers?

Architecturally, by giving the generator only retrieved context and instructing it to decline when the context does not contain the answer. That reduces the rate but does not eliminate it, so anything with real consequences needs the citations surfaced so a reader can check, rather than a confident paragraph on its own.

More templates