RAG pipeline architecture.
Almost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Client | Client | Supporting component |
| Query API | Application | Supporting component |
| Query rewriter | Application | Expands the question before retrieval |
| Retriever | Application | Hybrid: dense vectors plus BM25 |
| Reranker | Application | Cross-encoder over the top 50 |
| Generator | Application | Answers only from retrieved context |
| Ingest worker | Application | Chunk, embed, upsert |
| Embedding model | Application | Supporting component |
| Qdrant | Data | Supporting component |
| Document store | Data | Supporting component |
| Postgres | Data | Supporting component |
| Tracing | Infrastructure | Retrieval quality is only visible in traces |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Hybrid retrieval, because embeddings miss exact terms
Dense vectors are good at meaning and bad at specifics: part numbers, error codes, proper nouns and anything rare in the training distribution. Keyword search is the reverse. Running both and fusing the results costs a second index to maintain and a fusion parameter to tune, and it removes an entire class of failure where a user searches for the exact string in a document and gets nothing.
Reranking is where the quality is
Retrieving fifty candidates cheaply and then scoring them with a cross-encoder that reads query and document together is consistently better than retrieving ten and hoping. The cross-encoder cannot be precomputed, because it needs the pair, so it adds real latency proportional to the candidate count. That tradeoff, a slower answer that is right, is usually the correct one, and it is the single highest-leverage change in most underperforming RAG systems.
Chunking decides the ceiling and cannot be fixed later
Chunks that are too small lose the context that makes them interpretable; chunks that are too large dilute the embedding until it matches everything weakly. Whatever you pick is baked into the index, so changing it means re-embedding the entire corpus. This is the decision most worth prototyping properly before committing, and the one most often made by accepting a default.
Without tracing you are guessing
When an answer is wrong, the question is whether retrieval returned the wrong documents, the reranker ordered them badly, or the generator ignored what it was given. Those have completely different fixes and are indistinguishable from the output alone. Capturing the retrieved set, the scores and the final prompt for every request costs storage and adds a dependency, and it is the difference between improving the system and changing it at random.
How it changes with scale
Query volume drives the retrieval and reranking tiers, and reranking is the expensive one, so it is the first thing to cache or shrink. Corpus size drives the vector store, and vector search degrades gracefully until it does not: recall at a fixed latency drops as the index grows, which is a quality regression that no error rate will show you. Ingest is bursty by nature and belongs on its own workers so a bulk reindex cannot starve live queries.
Where it breaks first
Silent retrieval degradation. Nothing errors, latency looks normal, and answers slowly get worse as the corpus grows or drifts away from the queries people ask. Because there is no exception and no alert, this is usually discovered through user complaints months later, which is the argument for evaluating retrieval on a fixed question set continuously rather than only at launch.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
Do you need a dedicated vector database?
Not always. Postgres with pgvector is sufficient well past the point most teams assume, and it keeps your documents and their embeddings in one system with one backup story. A dedicated store earns its place when the index no longer fits comfortably in memory or when you need filtering and vector search to be fast at the same time.
How large should chunks be?
Large enough to stand alone as an answer to something, small enough that the embedding still means one thing. Splitting on document structure, headings and sections, beats splitting on a fixed token count, because the structure usually encodes exactly the boundary you want.
Can this run entirely locally?
Yes. Local embedding and generation models with a local vector store is a complete offline pipeline, which matters when the corpus is the sort of thing you cannot send to a third party. Quality is lower than frontier hosted models, though the gap on retrieval-grounded answering is much smaller than on open-ended generation.
How do you stop it inventing answers?
Architecturally, by giving the generator only retrieved context and instructing it to decline when the context does not contain the answer. That reduces the rate but does not eliminate it, so anything with real consequences needs the citations surfaced so a reader can check, rather than a confident paragraph on its own.
More templates
- E-commerce Platform ArchitectureFour services with four completely different consistency requirements, which is the whole reason this is not one application.
- Notification Service DesignThe architecture is straightforward. The reason this needs to be one service is that the rules about when not to send are the hard part, and they have to live somewhere every sender passes through.
- Distributed Rate Limiter DesignEvery design here is a trade between how accurate the limit is and how much latency you are willing to add to every single request to achieve it.
- Observability Stack ArchitectureThe architecture is mostly about cost control. Collecting everything is technically easy and financially ruinous, so the interesting decisions are all about what to throw away.
Last updated