Skip to main content

Design a Chat System

A chat system enables real-time text communication between individuals and groups. Platforms like WhatsApp, Slack, Discord, Microsoft Teams, and WeChat support billions of daily messages, persistent conversation history, and low-latency delivery across the globe. Designing such a system presents a fascinating set of challenges: managing millions of long-lived connections, ensuring messages are never lost or displayed out of order, and synchronizing state across multiple devices. This article provides a complete architectural walkthrough, from clarifying requirements to handling failure at scale.

Step 1: Requirements Clarification​

Functional Requirements​

  • One-to-one messaging – Two users can exchange text messages privately.
  • Group chat – Multiple users can participate in a shared conversation.
  • Message delivery – Messages are delivered in near real-time to online recipients.
  • Message persistence – Users can view historical messages, even after logging out and back in.
  • Online presence – The system shows whether a user is online or was last seen at a given time.
  • Read receipts – Senders know when their messages have been delivered and read.
  • Typing indicators – Recipients see when someone is typing.
  • Message history – Users can scroll back indefinitely and search old conversations.
  • File or media attachments – Users can send images, videos, and documents.
  • Multi-device synchronization – A user connected from a phone and a laptop sees the same conversation state.

Non-Functional Requirements​

  • Low latency – Message delivery should feel instantaneous, typically under a few hundred milliseconds.
  • High availability – The chat service must be reachable at all times; users tolerate very little downtime.
  • Reliable delivery – Messages must not be permanently lost, even if recipients are offline or servers restart.
  • Message ordering – Within a conversation, messages must appear in the order they were sent.
  • Scalability – The system must support millions of concurrent connections and billions of messages per day.
  • Offline support – Users who are offline must receive their messages as soon as they reconnect.
  • Fault tolerance – Failures of individual servers or network components should not cause data loss or extended unavailability.
  • Security and privacy – Data must be encrypted in transit; some applications additionally require end-to-end encryption.

Step 2: Capacity Estimation​

Let's assume the following scale:

  • Total users: 1 billion
  • Daily active users (DAU): 500 million
  • Average messages per user per day: 40
  • Total messages per day: 20 billion
  • Messages per second at peak: Assuming peak is 2x average: roughly 460,000 messages per second
  • Concurrent connections: 50 million persistent WebSocket connections
  • Storage: Each message averages 100 bytes of metadata, plus occasional media. For 20 billion messages/day, that is ~2 TB of text metadata daily, or ~730 TB per year.

Chat systems are both connection-heavy and write-heavy. The number of concurrent connections dictates the load on the WebSocket infrastructure, while the high message throughput demands careful database sharding and efficient disk I/O.

Step 3: API Design​

A chat system typically exposes a mix of REST endpoints for non-streaming operations and WebSocket for real-time messaging.

REST APIs​

  • Send message
    POST /api/v1/messages
    Body: { conversation_id, content, media_ids[], client_message_id }
    Returns: { message_id, server_timestamp }

  • Fetch conversation history
    GET /api/v1/conversations/{id}/messages?before={cursor}&limit=50
    Returns a paginated list of messages.

  • Mark message as read
    POST /api/v1/conversations/{id}/read
    Body: { last_read_message_id }

  • Get online presence
    GET /api/v1/users/{id}/presence
    Returns { status: "online" | "offline", last_seen: timestamp }

  • Upload attachment
    POST /api/v1/media
    Multipart upload returning a media_id and URL.

  • Create group chat
    POST /api/v1/conversations
    Body: { name, participant_ids[], type: "group" }

WebSocket Protocol​

Clients open a persistent WebSocket connection to receive messages and presence updates in real-time. The server can push frames such as:

  • new_message – Contains message content, sender, timestamp, and a server-assigned sequence number.
  • message_read – Indicates a message was read by another participant.
  • typing – Typing indicator event.
  • presence_change – A user came online or went offline.

Idempotency is achieved by having clients send a unique client_message_id; the server deduplicates messages with the same ID within a time window.

Step 4: High-Level Architecture​

The architecture separates the write path, the read path for history, and the real-time delivery path.

  • API Gateway: Handles REST traffic, authentication, rate limiting.
  • WebSocket Servers: Manage persistent connections from millions of devices.
  • Chat Service: Core business logic for sending, storing, and retrieving messages.
  • Presence Service: Tracks online/offline status and last-seen timestamps.
  • Message Queue: Decouples message persistence from real-time delivery; buffers incoming writes.
  • Message Database: Primary store for conversation history, sharded by conversation ID.
  • Notification Service: Sends push notifications (APNs/FCM) when users are offline.
  • Search Index: Enables full-text search over message history.

Step 5: Real-Time Message Delivery​

Real-time delivery relies on persistent WebSocket connections. When a client opens the app, it establishes a WebSocket to one of the WebSocket gateway servers. The gateway authenticates the user and registers the connection with the Presence Service.

For 1:1 messaging, when Alice sends a message to Bob:

  1. Alice’s client sends the message to her connected WebSocket server.
  2. The Chat Service validates the request, stores the message, and looks up Bob’s active connections from the Presence Service.
  3. If Bob is online, the message is pushed to Bob’s WebSocket server, which forwards it to Bob’s client.
  4. If Bob is offline, the message is stored and a push notification is dispatched via the Notification Service.

For group messaging, the Chat Service retrieves all online members of the group and fans out the message to each of their WebSocket connections. For very large groups, fan-out is performed asynchronously using a message queue, and delivery may take slightly longer.

Step 6: Message Storage Design​

Messages are the core data entity. A typical message record contains:

{
  "message_id": "msg_abc123",
  "conversation_id": "conv_xyz",
  "sender_id": "user_456",
  "content": "Hello!",
  "media_refs": [],
  "timestamp": 1700000000000,
  "client_message_id": "client_789"
}

Partitioning: The primary access pattern is retrieving messages for a specific conversation, ordered by time. Therefore, the Message Database is sharded by conversation_id. Within each shard, messages are stored in a time-ordered table or wide-column store.

Read/write optimization: Writes are append-only and highly concurrent. Reads are sequential scans of recent messages. Caching the most recent messages of active conversations in Redis dramatically reduces database load.

SQL vs NoSQL:

  • NoSQL (e.g., Cassandra, DynamoDB) excels at high-throughput writes, time-series data, and horizontal scaling with conversation-based partitioning.
  • SQL (e.g., PostgreSQL) provides richer querying and transactional guarantees but requires careful sharding at scale.

Many large chat systems prefer NoSQL for the message store and keep user/group metadata in a relational database.

Step 7: Message Ordering and Delivery Guarantees​

Maintaining in-order delivery within a conversation is crucial, but ordering is guaranteed only within a single shard or partition. For a given conversation, every message is written to the same partition, and the storage engine assigns a monotonically increasing sequence number or uses a server-generated timestamp (with conflict resolution).

Delivery semantics in most chat systems are at-least-once: a message may be delivered more than once due to retries. The client must be idempotent—it uses client_message_id to deduplicate and ignores a message if it has already been processed.

Exactly-once processing is theoretically extremely difficult in distributed systems; practical implementations achieve "effectively once" through deduplication and idempotent operations. Undeliverable messages are routed to a dead letter queue after exhausting retries, where they can be inspected and replayed manually.

Step 8: Presence and Typing Indicators​

The Presence Service must track whether a user is currently connected. This is implemented using heartbeats: clients send a periodic pulse over the WebSocket. If a server detects a missing heartbeat for a configurable timeout, it marks the user as offline and broadcasts the status change to relevant contacts.

Typing indicators are ephemeral signals. They are sent over the WebSocket frame without persisting to the database. To avoid flooding, clients throttle typing indicators to at most one every few seconds. The Presence Service does not need to store typing state—it simply forwards the event.

Trade-off: Accurately reflecting online status for millions of users requires a highly available, in-memory data store with fast lookups (e.g., Redis). The system sacrifices perfect accuracy for scalability: a user is shown as online if any of their devices has sent a recent heartbeat; brief network glitches may cause a few seconds of stale presence.

Step 9: Group Chat Design​

Group chats add complexity because a single message must be delivered to many recipients.

  • Small groups (up to a few hundred members): On each new message, the Chat Service looks up the member list, retrieves online connections from the Presence Service, and fans out the message directly to each recipient's WebSocket server.
  • Large groups (thousands to millions of members): Full synchronous fan-out would overwhelm the service. Instead, the message is written to a shared "channel" that members subscribe to. Clients periodically pull new messages from the channel, or the server pushes messages to a dedicated message queue per user, which is consumed asynchronously.

Membership changes are stored in a group metadata service. Adding or removing a member updates the fan-out list and, for very large groups, may trigger a rebalance of subscription channels.

Step 10: Offline Messaging and Multi-Device Sync​

When a user is offline, incoming messages are stored in the Message Database as usual. Upon reconnection, the client fetches missed messages using the last received message_id or server timestamp.

Multi-device sync means a user on a laptop sees messages sent from their phone. The Chat Service associates a user’s multiple devices and delivers a copy of each message (or a notification) to all active devices. The message store serves as the single source of truth; each device independently fetches history when it comes online.

Push notifications (via APNs or FCM) wake up the mobile app when a new message arrives while the app is in the background.

Step 11: Media Attachments​

For images, videos, and files, the upload flow is separated from the message flow:

  1. Client uploads the file via a REST endpoint to a dedicated media service or directly to object storage (using a pre-signed URL).
  2. A media processing pipeline performs virus scanning, thumbnail generation, and transcoding into multiple resolutions.
  3. The processed media is stored in Object Storage (e.g., Amazon S3) and served through a CDN.
  4. The upload returns a media_id, which the client then includes in the message payload.
  5. When a recipient views the message, they download the media directly from the CDN, offloading bandwidth from the chat servers.

Step 12: Search and Message Indexing​

Users need to search their conversation history by keyword. A Search Index (e.g., Elasticsearch) is populated asynchronously from the Message Database via a change data capture (CDC) pipeline. The index is sharded by user ID, so a search query is limited to the user’s own messages.

Indexing introduces a slight delay between message send and searchability, which is acceptable for most chat applications. Search infrastructure is scaled independently from the real-time chat path.

Step 13: Scalability Strategies​

  • Horizontal scaling: WebSocket servers and Chat Service instances are stateless and scale horizontally behind load balancers.
  • Sharding: Message database sharded by conversation ID. Presence service sharded by user ID.
  • Caching: Redis caches hot conversations, online status, and user metadata.
  • Asynchronous processing: Fan-out for large groups, media processing, and search indexing are handled by message queues, decoupling critical real-time paths from heavy backend work.
  • Geo-distribution: Deploy WebSocket gateways and chat services in multiple regions. Route users to the nearest data center. Cross-region communication is handled by replicating message data asynchronously or by routing cross-region conversations through a single home region.

Step 14: Reliability and Failure Handling​

  • Connection drop handling: Clients automatically reconnect with exponential backoff. On reconnect, they sync missed messages.
  • Retry logic: Failed message deliveries are retried with exponential backoff and capped retry counts. Idempotency prevents duplicates.
  • Backpressure: If the message queue or database is overwhelmed, the WebSocket gateway applies backpressure by rejecting new messages with a "service overloaded" status, asking clients to retry later.
  • Failover: WebSocket connections are sticky to a particular server. If a server fails, clients reconnect to a different server, and the Presence Service updates accordingly.
  • Graceful degradation: If presence or typing indicators fail, core messaging continues to function. If the CDN is slow, clients can fall back to a smaller thumbnail.
  • Message queue buffering: Kafka or an equivalent distributed log ensures messages are not lost even if downstream consumers (like the search indexer) are temporarily down.

Step 15: Security and Privacy​

  • Authentication: All connections are authenticated using short-lived tokens (JWT) or session cookies. WebSocket connections are authenticated on upgrade.
  • Authorization: A user can only send messages to conversations they are a member of. Group membership is verified on every operation.
  • End-to-end encryption (E2EE): While full E2EE implementation is complex, the architecture must support it conceptually. In an E2EE model, messages are encrypted on the sender's device and decrypted only on the recipient's devices. The server stores only ciphertext and cannot read message content. Group key management and key rotation are additional challenges.
  • Encryption at rest: Message databases and media storage are encrypted using AES-256 or equivalent.
  • Abuse prevention: Rate limiting per user and per IP prevents spam. Automated moderation tools and user reporting handle abusive content.
  • Moderation: Large platforms employ machine learning models and human review queues to detect and remove policy-violating messages.

Real-World Example: WhatsApp-like Chat​

Consider a WhatsApp-style application supporting 1:1 and group messaging, delivery receipts, read receipts, and offline delivery.

  1. Alice sends a message; the server persists it immediately and publishes an event.
  2. If Bob is online, the message is pushed in real-time. Bob's client acknowledges delivery.
  3. If Bob is offline, a push notification wakes his app, which then fetches missed messages.
  4. When Bob reads the message, a read receipt propagates back to Alice.

Trade-offs​

  • Latency vs consistency: Strict global ordering requires coordination and slows delivery. Most chat systems guarantee ordering per conversation (single partition) and accept slight delays for cross-region synchronization.
  • Ordering vs scalability: Global ordering across all users would require a single sequencer and limit scalability. Per-conversation ordering is a practical compromise.
  • Storage cost vs history retention: Storing every message forever is expensive. Some services limit history or archive old messages to cold storage.
  • Real-time delivery vs offline reliability: Prioritizing real-time delivery for online users should not compromise reliable storage for offline retrieval. The architecture must treat the message database as the source of truth.
  • Simplicity vs feature richness: Every feature (typing indicators, read receipts, emoji reactions) adds state and traffic. Each should be evaluated against its impact on scalability and complexity.

Common Mistakes​

  • Using HTTP polling only – Inefficient and cannot achieve the low latency required for chat. WebSockets or similar persistent connections are essential.
  • Ignoring offline support – Users frequently go in and out of connectivity. A robust store-and-forward mechanism is non-negotiable.
  • No idempotency – Network retries cause duplicate messages. Clients must deduplicate, and servers should guard against double-processing.
  • No message ordering strategy – Simply relying on timestamps risks out-of-order display due to clock skew. Sequence numbers per conversation are safer.
  • Overloading the presence service – Updating presence on every tiny status change overwhelms the system. Use heartbeats and cooldown periods.
  • Not planning for fan-out – A single message to a large group can multiply into millions of deliveries. Use asynchronous fan-out and group-specific optimizations.
  • Storing attachments in the database – Blobs in a relational database perform poorly. Use object storage and a CDN.

Interview Perspective​

In a system design interview, the chat system question tests your ability to handle real-time, stateful systems. Expect to discuss:

  • How do you deliver messages in real time? (WebSocket, event-driven delivery)
  • How do you handle offline users? (Store and forward, push notifications)
  • How do you ensure ordering? (Per-conversation sequence numbers, sharding)
  • How do you scale group chat? (Async fan-out, large-group channels)
  • How do you store messages? (Sharded NoSQL or SQL, with caching)
  • How do you support multiple devices? (User account linked to multiple connections, sync on reconnect)

Demonstrate that you can balance real-time performance with data durability, and that you understand the cost of each design choice.

Summary​

Designing a chat system requires orchestrating persistent WebSocket connections, reliable message storage, ordered delivery, presence tracking, and offline synchronization. The architecture separates concerns across a WebSocket gateway for real-time delivery, a sharded message database for durability, a message queue for asynchronous processing, and a notification service for offline reach.

A production chat system is a careful blend of real-time communication, distributed data stores, and event-driven processing. By applying the patterns and trade-offs discussed here, you can design a chat platform that scales to billions of messages while remaining fast, reliable, and intuitive.

Further Reading​