System design template

Feature flag service architecture.

Evaluation has to be local and instant, because a flag check sits in the hot path of code that would otherwise not make a network call at all.

Download for macOS
v0.1.33 · .dmg · Apple Silicon & Intel
Signed & notarized by Apple · opens without a Gatekeeper warning
sha256 698955a0187bc039f4c74f5d05a9f10fbb27376a45788a0a241d1326b73873c7
Download for Windows instead
$curl -fsSL https://lucidtrain.com/install.sh | sh
Feature flag service architecture. 11 components across 4 tiers.
Hover any component to see what it is responsible for.
Feature flag service architecture. Rendered by the same ELK layout engine the app runs: the agent emits components, tiers and edges, and the engine places them, so the boxes cannot overlap.

The components

Every row below is read from the graph that produced the diagram above, so the two cannot disagree.

ComponentTierWhy it is there
Client SDKsClientEvaluate locally, never block on a network call
CDNEdgeServes the ruleset, cached at the edge
Streaming endpointEdgeSSE push, so a kill switch is instant
Flag APIApplicationSupporting component
Rule compilerApplicationTargeting rules to a payload SDKs can evaluate
Audit logApplicationWho flipped what, and when
Exposure eventsApplicationWhich variant each user actually saw
PostgresDataSupporting component
RedisDataSupporting component
KafkaDataSupporting component
WarehouseDataExperiment analysis, not the serving path

Design decisions worth arguing about

A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.

Evaluate in the SDK, never over the network

A flag check happens in code paths that run thousands of times a second, so a network call per evaluation is out of the question. The SDK holds the compiled ruleset in memory and evaluates locally in microseconds. The cost is that every client is briefly out of date after a change, and that the ruleset must be small enough to hold and cheap enough to evaluate.

Streaming, because a kill switch is the point

Polling every thirty seconds is fine for a gradual rollout and useless for the case the product actually exists to serve, which is turning something off immediately during an incident. A streaming connection pushes changes in under a second. The cost is holding an open connection per client process and a fallback path for networks that will not permit it.

The CDN is the availability story

If the flag service is down and SDKs cannot fetch a ruleset, applications either fail or run on stale defaults. Serving the ruleset as a static artifact from a CDN means the read path survives an outage of everything else you operate. It also means a change is not live until the cache expires or is purged, so purge behaviour becomes load-bearing.

Exposure events are a separate system

Knowing which variant a user actually saw is what makes experiments analysable, and it is high-volume telemetry with nothing in common with flag serving. Keeping it on its own path means an analytics outage cannot affect evaluation. The consequence is that experiment results lag, sometimes by hours, which surprises people expecting a live dashboard.

How it changes with scale

Evaluation cost is zero to the service, since it happens in the client. What scales with usage is exposure event volume, which grows with traffic times flag count and quickly exceeds every other write in the system. Ruleset size grows with flag count and matters because it is held in memory in every process.

Where it breaks first

A flag that never gets cleaned up. The technical system stays healthy while the ruleset accumulates hundreds of permanently-on flags, each one a branch in the code nobody removes. The failure is organisational and shows up as evaluation latency and unreadable code rather than as an outage, which is why flag expiry dates and an owner per flag matter more than any part of this diagram.

Draw this yourself

Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.

shell
$ Diagram a feature flag service: client SDKs evaluating locally, a CDN serving the ruleset, an SSE streaming endpoint for instant updates, a rule compiler, exposure events into Kafka and a warehouse.

When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.

FAQ

Questions about this design

Should flag evaluation happen server-side or in the client?

In the SDK, wherever it runs, using a ruleset fetched in advance. A network call per evaluation is too slow for the code paths flags actually sit in.

How do you avoid leaking upcoming features to browser clients?

You cannot fully: a ruleset in the browser is readable. Anything genuinely sensitive should be evaluated server-side and the client told only the outcome.

Build or buy?

Buy, unless you have unusual requirements. The serving path is simple; the parts that take time are the SDKs for every language you use and the experiment analysis on the other end.

How long should a flag live?

Give it an expiry date when it is created and treat a passed date as a bug. Flags that outlive their rollout are the main long-term cost of the whole system.

More templates