System design template

Vector search service architecture.

The failure mode here is not an error. It is recall dropping slowly as the index grows, with nothing in your metrics saying so.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
Vector search service architecture. 12 components across 4 tiers.
Hover any component to see what it is responsible for.
Vector search service architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
ClientClientSupporting component
Search APIApplicationSupporting component
Embedding serviceApplicationSame model at index and query time or recall collapses
Hybrid retrieverApplicationDense vectors plus BM25, fused
Filtered searchApplicationPre-filter or post-filter changes recall a lot
RerankerApplicationSupporting component
Index builderApplicationRebuilds and swaps atomically
QdrantDataHNSW, recall degrades quietly as it grows
ElasticsearchDataSupporting component
PostgresDataSupporting component
Object storageDataSupporting component
PrometheusInfrastructureTrack recall on a fixed query set, not just latency

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

The same embedding model at index and query time

Vectors are only comparable within the model that produced them, so changing the embedding model means re-embedding the entire corpus before a single query is served. Treating the model version as part of the index identity, and refusing to query an index built with a different one, turns a silent recall collapse into a loud error. Teams learn this once.

Hybrid retrieval, because dense vectors miss exact terms

Embeddings are good at meaning and poor at specifics: part numbers, error codes, rare proper nouns. Keyword search is the reverse. Running both and fusing costs a second index to maintain and a fusion weight to tune, and it removes a whole class of failure where a user searches for a literal string that is in the corpus and gets nothing.

Pre-filter or post-filter changes recall more than people expect

Filtering after the vector search is simple and can return far fewer results than requested, because the nearest neighbours were all filtered out. Filtering during search preserves recall and requires index support for it. The naive implementation looks correct in testing on a small corpus and degrades badly with selective filters at scale.

Rebuild and swap atomically

Mutating a live index means locking or serving inconsistent results mid-update. Building a new one, validating it against a fixed query set, and swapping the pointer makes updates instant and lets you refuse to ship an index whose recall regressed. It costs double the memory during the swap, which is the price of never serving a half-built index.

How it changes with scale

Query volume drives the reranker, which is the expensive stage and the first thing to cache. Corpus size drives the vector index, and approximate nearest neighbour search degrades gracefully in latency while degrading recall invisibly, which is the asymmetry that matters. Filter cardinality is the third axis and the one most often forgotten in capacity planning.

Where it breaks first

Silent recall degradation. Nothing errors, latency looks normal, and the right document stops being returned as the index grows or drifts away from the queries people ask. Because there is no exception and no alert, this is found through user complaints months later, which is the argument for evaluating recall on a fixed labelled query set on every index build rather than only at launch.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram a vector search service: embedding service, hybrid retrieval over a vector store and a keyword index, filtered search, a reranker, an index builder that swaps atomically, and metrics.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Do you need a dedicated vector database?

Not always. Postgres with pgvector is sufficient well past the point most teams assume. A dedicated store earns its place when the index no longer fits comfortably in memory or when you need selective filtering and vector search to be fast simultaneously.

How do you measure whether search is working?

Recall at k on a fixed labelled query set, evaluated on every index build. Latency and error rate tell you nothing about whether the right documents came back.

Is reranking worth the latency?

Usually yes. Retrieving fifty candidates cheaply and scoring them with a cross-encoder is consistently better than retrieving ten and hoping, and it is the single highest-leverage change in most underperforming retrieval systems.

How do you handle updates to a single document?

Upsert into the live index for freshness, and rely on the periodic rebuild for correctness. Incremental updates to approximate indexes degrade their structure over time, which is another reason recall needs measuring rather than assuming.

More templates