System design template

Observability stack architecture.

The architecture is mostly about cost control. Collecting everything is technically easy and financially ruinous, so the interesting decisions are all about what to throw away.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
Observability stack architecture. 11 components across 6 tiers.
Observability stack architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
Instrumented servicesClientOpenTelemetry SDK, one wire format
OTel collectorEdgeSampling and redaction happen here, before egress
PrometheusApplicationSupporting component
LokiApplicationIndexes labels, not log bodies
Trace backendApplicationSupporting component
AlertmanagerApplicationRoute on symptoms, not on causes
SLO evaluatorApplicationError budget burn rate, not raw thresholds
Metric storageDataSupporting component
Log storageDataSupporting component
GrafanaInfrastructureSupporting component
PagingExternalSupporting component

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

One collector tier, so sampling and redaction happen once

Letting every service ship directly to every backend means sampling policy and secret redaction are reimplemented per service and per language. A collector in front gives you one place to change both, and one place where an accidental log of a token can be stopped before it leaves your network. It is another hop to run and another thing that can drop data, which is why it should be the boring, well-monitored component in the diagram.

Index labels, not log bodies

Full-text indexing every log line is what makes observability bills alarming. Indexing only a small set of labels and brute-forcing the body at query time inverts the tradeoff: ingestion is cheap, and queries are slower but usually scoped to a time range you already know. The cost is that unstructured searches across everything are painful, so label design stops being cosmetic and becomes the main thing determining whether the system is usable.

Alert on symptoms, not causes

Alerting on CPU, memory and queue depth produces pages for conditions users never notice and misses outages that show up in none of them. Alerting on error budget burn means you are paged when the thing you promised is actually at risk. It requires defining an SLO, which is a product conversation people avoid, and that avoidance is why most alerting is cause-based.

Tail sampling costs more and keeps the useful traces

Sampling at the head, deciding at the start of a request, is cheap and throws away the slow and failing traces at the same rate as the boring ones, which are exactly the traces you wanted. Deciding at the tail, once the outcome is known, keeps the interesting ones, at the cost of buffering every span until the trace completes. That buffer is real memory in the collector and is the main operational cost of doing this properly.

How it changes with scale

Cost grows with cardinality far faster than with traffic. One badly chosen label containing a user id or a request id can multiply metric series by millions and take down the metrics backend, which is why cardinality limits belong in the collector rather than in a code review guideline. Log and trace volume grow with traffic and are controlled by retention and sampling, both of which are much easier to set before the bill arrives.

Where it breaks first

The observability stack failing at the same time as the system it observes, usually because they share infrastructure. Losing visibility exactly when you need it turns a short incident into a long one. This is the argument for the alerting path in particular being independent of the primary cluster, even at the cost of running it somewhere less convenient.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram an observability stack: OpenTelemetry-instrumented services, a collector doing sampling and redaction, Prometheus for metrics, Loki for logs, a trace backend, SLO evaluation and alerting, with Grafana on top.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Do you need traces if you already have metrics and logs?

In a distributed system, yes. Metrics tell you something is slow and logs tell you what one service did; only traces tell you where the time went across service boundaries. In a monolith the case is much weaker and a profiler often serves better.

Why OpenTelemetry rather than a vendor SDK?

It decouples instrumentation from backend. Instrumenting with a vendor SDK means changing backends is a code change across every service, which is precisely the leverage the vendor is relying on.

How long should you keep telemetry?

Different answers per signal. Metrics compress well and stay useful for capacity planning, so months to years downsampled. Traces and logs are mostly used within days of being written, so short retention with the option to keep a sampled subset longer.

Can this run locally against a local model?

The stack itself runs locally without difficulty, which is how most people develop against it. If you want an agent querying it, that agent can also be local, which matters because logs and traces routinely contain data you would rather not send to a third party.

More templates