Real-time chat architecture.
Long-lived connections change everything: the hard part is not storing messages, it is knowing which of your servers is holding the socket you need to write to.
The components
Every row below is read from the graph that produced the diagram above, so the two cannot disagree.
| Component | Tier | Why it is there |
|---|---|---|
| Mobile & web | Client | Supporting component |
| Load balancer | Edge | Sticky, so a socket stays on one node |
| WebSocket gateway | Application | Holds the long-lived connections |
| Presence service | Application | Supporting component |
| Message service | Application | Ordering and dedup live here |
| Fan-out worker | Application | Supporting component |
| Pub/sub | Data | Routes between gateway nodes |
| Kafka | Data | Durable log, replay on reconnect |
| Cassandra | Data | Partitioned by conversation |
| Object store | Data | Attachments, never through the socket |
| APNs / FCM | External | For offline recipients |
Design decisions worth arguing about
A diagram shows what was chosen. It does not show what it cost, and that is usually the part that matters in a review or an interview.
Stateful gateways force sticky routing
A WebSocket is a connection pinned to one process, which makes the gateway tier stateful in a way ordinary request handling is not. Delivering a message means finding the node holding the recipient's socket, which is what the pub/sub layer is for. The consequence people underestimate is deployment: rolling the gateway tier disconnects every client on the node being replaced, so reconnect behaviour, backoff, and resuming from the last received message stop being edge cases and become the normal path.
Ordering is per conversation, not global
Guaranteeing a total order across the whole system is expensive and nobody needs it. Guaranteeing that messages within one conversation arrive in a consistent order is both cheaper and what users actually perceive as correct. Partitioning the log by conversation id gives you that for free, but it also means a single extremely busy conversation is a single partition, and a single partition is a single consumer's throughput ceiling.
Attachments never travel through the socket
Pushing file bytes over the same connection that carries messages means one upload can stall an entire conversation, because it is one ordered stream. Uploading directly to object storage and sending only a reference keeps the message path small and predictable. The price is a second failure mode to handle in the client: a message that references a file that has not finished uploading.
Presence is expensive and mostly a lie
Accurate presence means every client heartbeating constantly, and the write volume from heartbeats can exceed the message volume by a wide margin. Most systems degrade deliberately: coarse states rather than exact ones, a generous timeout before marking someone away, and no attempt to be correct during a network partition. Users tolerate stale presence far better than they tolerate delayed messages, so this is the right thing to spend consistency on.
How it changes with scale
Connection count drives the gateway tier and it scales close to linearly, since each connection costs memory more than CPU. Message volume drives the log and the fan-out workers. The two grow independently, which is the main argument for keeping them in separate tiers: a product with many idle users and few messages has a completely different bill from one with the reverse.
Where it breaks first
A reconnect storm. Anything that drops a large number of sockets at once, a deploy, a load balancer restart, an upstream network blip, causes every client to reconnect at roughly the same moment, and each reconnect is more expensive than a steady-state message because it re-establishes the connection and replays missed history. Without jittered backoff the recovery attempt is heavier than the original load and the system does not come back on its own.
Draw this yourself
Open the Diagram tab and describe the system. The agent emits a semantic graph rather than coordinates, so you can edit the components and the layout re-solves instead of drifting.
When the shape is right, Implement in code turns the canvas into a markdown specification, every component, every relationship and the notes, and starts a real turn in the Code tab with it.
Questions about this design
WebSockets or long polling?
WebSockets for anything conversational. Long polling still has a place as a fallback where corporate proxies interfere with upgrades, and it is worth keeping that path working rather than assuming every network allows a socket.
Why Kafka as well as a database?
They answer different questions. The database answers what the history is; the log answers what happened recently and in what order, and lets a client that was offline replay from a known position. You can build chat with only a database, but resuming after a disconnect gets considerably harder.
How are messages delivered to offline users?
The fan-out worker checks presence, and if the recipient has no live connection it hands off to APNs or FCM instead. The subtlety is deduplication: a user who reconnects mid-delivery can receive both the socket message and the push, so the client needs stable message ids.
Does this design support end-to-end encryption?
The shape survives, but the server-side features do not. Search, moderation and rich push previews all assume the server can read the message, and with end-to-end encryption none of them can. That is a product decision long before it is an architecture one.
More templates
- Payment System DesignThe only system on this list where being approximately right is indistinguishable from being wrong, and where the provider, not you, holds the truth.
- RAG Pipeline ArchitectureAlmost every RAG system that disappoints is failing at retrieval, not generation, and the architecture is what decides whether you can tell.
- E-commerce Platform ArchitectureFour services with four completely different consistency requirements, which is the whole reason this is not one application.
- Notification Service DesignThe architecture is straightforward. The reason this needs to be one service is that the rules about when not to send are the hard part, and they have to live somewhere every sender passes through.
Last updated