Real-time chat architecture.
Long-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Mobile & web | Client | Supporting component |
| Load balancer | Edge | Sticky, so a socket stays on one node |
| WebSocket gateway | Application | Holds the long-lived connections |
| Presence service | Application | Supporting component |
| Message service | Application | Ordering and dedup live here |
| Fan-out worker | Application | Supporting component |
| Pub/sub | Data | Routes between gateway nodes |
| Kafka | Data | Durable log, replay on reconnect |
| Cassandra | Data | Partitioned by conversation |
| Object store | Data | Attachments, never through the socket |
| APNs / FCM | External | For offline recipients |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Stateful gateways force sticky routing
A WebSocket is a connection pinned to one process, which makes the gateway tier stateful in a way ordinary request handling is not. Delivering a message means finding the node holding the recipient's socket, which is what the pub/sub layer is for. The consequence people underestimate is deployment: rolling the gateway tier disconnects every client on the node being replaced, so reconnect behaviour, backoff, and resuming from the last received message stop being edge cases and become the normal path.
Ordering is per conversation, not global
Guaranteeing a total order across the whole system is expensive and nobody needs it. Guaranteeing that messages within one conversation arrive in a consistent order is both cheaper and what users actually perceive as correct. Partitioning the log by conversation id gives you that for free, but it also means a single extremely busy conversation is a single partition, and a single partition is a single consumer's throughput ceiling.
Attachments never travel through the socket
Pushing file bytes over the same connection that carries messages means one upload can stall an entire conversation, because it is one ordered stream. Uploading directly to object storage and sending only a reference keeps the message path small and predictable. The price is a second failure mode to handle in the client: a message that references a file that has not finished uploading.
Presence is expensive and mostly a lie
Accurate presence means every client heartbeating constantly, and the write volume from heartbeats can exceed the message volume by a wide margin. Most systems degrade deliberately: coarse states rather than exact ones, a generous timeout before marking someone away, and no attempt to be correct during a network partition. Users tolerate stale presence far better than they tolerate delayed messages, so this is the right thing to spend consistency on.
How it changes with scale
Connection count drives the gateway tier and it scales close to linearly, since each connection costs memory more than CPU. Message volume drives the log and the fan-out workers. The two grow independently, which is the main argument for keeping them in separate tiers: a product with many idle users and few messages has a completely different bill from one with the reverse.
Where it breaks first
A reconnect storm. Anything that drops a large number of sockets at once, a deploy, a load balancer restart, an upstream network blip, causes every client to reconnect at roughly the same moment, and each reconnect is more expensive than a steady-state message because it re-establishes the connection and replays missed history. Without jittered backoff the recovery attempt is heavier than the original load and the system does not come back on its own.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
More templates
- URL Shortener System DesignThe canonical read-heavy system: roughly a hundred reads for every write, and a redirect that has to be fast enough that nobody notices it happened.
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.
- E-commerce Platform ArchitectureFour services with four completely different consistency requirements, which is the whole reason this is not one application.