Build a messaging platform where a message typed on one phone shows up on another phone anywhere in the world within a fraction of a second - reliably, in order, end-to-end encrypted, and even when the recipient is offline.
End-to-end encryption (E2EE) means messages are encrypted on the sender's phone and decrypted only on the recipient's phone. The server passes along ciphertext - scrambled bytes it has no key for. That one constraint quietly reshapes the whole system. The server cannot search message content, cannot scan it for moderation, and cannot show your chat history on a new phone, because it never sees any content in readable form. It also means the server has little reason to keep messages at all: once a message is delivered to every recipient, the server deletes its copy. The server is a relay, not an archive - your chat history lives on your devices, not in our database. That is the opposite of most systems in this practice set, and it is why the storage estimates below come out surprisingly small. What the server still sees is metadata: who messaged whom, when, and how big the message was. It needs some of that to route and deliver. Treating metadata as sensitive (minimal retention, minimal logging) is part of taking E2EE seriously, and worth saying in an interview.
Each check mark is an acknowledgment traveling back along the path the message took. One check (sent) means the server has durably stored the message - if your phone dies right now, the message still exists. Two checks (delivered) means the recipient's phone has it - the server's job is done and it can delete its copy. Blue checks (read) mean the recipient actually opened the conversation. Each is a tiny message flowing backwards, and they ride the exact same delivery pipeline as normal messages. Ordering is promised per conversation, not globally. Within one chat, everyone sees messages in the same order with no gaps: message 1248 always comes after 1247. Across different chats there is no promise at all - a message in your family group and one from a coworker have no defined order between them, and nobody would ever notice. That narrow promise is a gift to the architecture: because ordering only matters within a conversation, we can shard the system by conversation and let each shard order its own messages independently, with no coordination between shards. A global ordering guarantee would force every message in the world through some agreement step - vastly more expensive, for a property no user can even observe.
The numbers say something unusual: 1.2M writes/sec sounds like a storage nightmare, but because the server deletes every message once it is delivered, almost nothing accumulates. The hard problem is not where to keep 100B messages - it is how to move 100B messages/day to the right phones, in order, without losing any. The design should be judged as a delivery pipeline, not a database. What storage remains splits three ways. In-flight messages (sent but not yet delivered to everyone) go to Cassandra, partitioned by conversation_id and ordered by a sequence number within each partition: appends are cheap, and "give me everything in this chat after sequence N" is a single ordered range read - exactly the query an offline device asks when it reconnects. This is the same append-and-read-forward shape as the change feed in Design Dropbox. Ephemeral state - who is online, which gateway holds each device's connection, typing indicators - lives in Redis with TTLs. It is rewritten constantly and losing it costs nothing but a moment of staleness, the same "disposable data gets the cheap path" argument as presence in Design Excalidraw. Small relational truth - accounts, group membership, public key registry - lives in PostgreSQL. It is modest in size, changes rarely, and wants constraints (one account per phone number). Media takes the Dropbox route entirely: encrypted blobs go straight to object storage via presigned URLs, downloads come from CDN edges, and the message itself carries only a pointer and a decryption key. The 2PB/day of media never touches the messaging pipeline. The contrast worth naming: Design a Newsfeed stores content once and serves it to readers forever; WhatsApp delivers each message once to a known small set of devices and then forgets it. Store-and-serve versus route-and-delete - they lead to completely different systems.
SEND - one-on-one message from A to B
[A's phone] ==WebSocket==> [Gateway 12]
|
v
[Chat Service - owner of conv_42]
assign seq 1248 -> store in Message Store
|
ack back to A ("sent" - one check)
|
v
[Session Registry (Redis)] -- where is B?
| |
B online B offline
| |
v v
[Gateway 87] [Inbox (B's queue)]
==push==> B +
| [Push notification "wake up"]
B's phone acks |
| B reconnects later ->
v drain inbox in order -> ack each
"delivered" (two checks) flows back to A
Message Store deletes the delivered message
GROUP - one message, many inboxes
[Chat Service] -> [Group Service] members? -> write pointer to each
member's Inbox -> push to the online ones via their
gateways (bounded: groups cap at ~1,000 members)
MEDIA - bytes never touch the messaging pipeline
[A] --presigned upload--> [Blob Storage (client-encrypted)] --> [CDN]
the message carries only a pointer + decryption key; B downloads
from the CDN edge (same direct-transfer pattern as Design Dropbox)With 200M open connections spread over ~3,000 gateways, delivery starts with a lookup problem: which gateway is holding the recipient's socket right now? The Session Registry answers it. When a device connects, its gateway writes "user u_B is on gateway 87" into Redis; when the socket drops, the entry is cleared (with a TTL as the safety net for crashed gateways). Delivering a message means one registry read, then a hop to that gateway, which pushes the message down the open socket. Compare this with Design Excalidraw, which routes sticky by board so that everyone editing a board lands on the same server and shares its memory. Chat cannot do that - you and your contact should not need to share a server, because you each talk to many people. So instead of co-locating conversation participants, WhatsApp keeps a directory of where everyone is and routes hop-by-hop: registry lookup, gateway-to-gateway hop, socket push. Sticky routing moves people to the state; a session registry moves messages to the people. Knowing which one fits which problem is a genuinely useful interview instinct.
Because delivery is the product, not storage. A message exists on the server only for the gap between "sender's phone sent it" and "every recipient's phone has it" - typically milliseconds, at most 30 days. Two forces make this the right call. First, E2EE makes stored messages nearly useless to us: we cannot search them, index them, or render them on a new device - they are opaque ciphertext. Keeping them would be all liability (a growing pile of intercepted-looking data) and no feature. Second, deleting-on-delivery turns the scaling problem from "store 36 trillion messages a year" into "hold roughly one hour's worth of undelivered traffic," which fits in a few terabytes. The Message Store behaves like a buffer, not a library. The Inbox follows the Newsfeed pointer trick: entries are (conversation, seq) references, ~50 bytes each, never copies of content. One message to a 1,000-member group writes one message row and 1,000 tiny pointers - and when each member's phone acknowledges, its pointer is deleted, the ref-count drops, and eventually the message row itself goes. Small, self-cleaning structures everywhere.
Database Schemas
- Users (PostgreSQL)
user_id UUID PRIMARY KEY,
phone_number VARCHAR(20) UNIQUE NOT NULL, -- one account per number: a DB rule
display_name VARCHAR(100),
device_tokens JSONB, -- push notification handles per device
public_keys JSONB, -- registered per-device public key bundles
last_seen_policy policy_enum DEFAULT 'contacts',
created_at TIMESTAMP DEFAULT NOW()
- Conversations (PostgreSQL)
conversation_id UUID PRIMARY KEY,
type conv_type_enum NOT NULL, -- 'direct' | 'group'
name VARCHAR(100), -- groups only
created_by UUID REFERENCES users(user_id),
created_at TIMESTAMP DEFAULT NOW()
- Group Members (PostgreSQL)
conversation_id UUID NOT NULL,
user_id UUID NOT NULL,
role member_role_enum DEFAULT 'member', -- 'member' | 'admin'
joined_at_seq BIGINT, -- membership starts at this sequence number
PRIMARY KEY (conversation_id, user_id)
INDEX idx_member_convs ON (user_id) -- "which chats am I in?"
- Messages (Cassandra) -- IN-FLIGHT ONLY: deleted once fully delivered
PRIMARY KEY ((conversation_id), seq)
Fields: message_id UUID, sender_id, ciphertext BLOB,
msg_type ('text'|'media'|'system'), media_pointer, sent_at,
recipients_pending INT -- delivered-to-all when this hits 0
TTL: 30 days (the offline-retention backstop)
-- partition per conversation = one ordered strip of data
-- clustering by seq makes "everything after N" a single range read
-- (the old design keyed this by message_id, which cannot answer
-- either ordering or catch-up queries - the partition key IS the design)
- Inbox (Cassandra) -- per-recipient pointers to undelivered messages
PRIMARY KEY ((user_id), conversation_id, seq)
Fields: enqueued_at
TTL: 30 days
-- ~50-byte pointers, never message copies (the Newsfeed pointer move)
-- drained in order on reconnect; each ack deletes its row
Redis keys
- Session Registry + Presence
Key: "sess:{user_id}" -> { "gateway": "gw-87", "online": true,
"last_seen": "2026-07-05T18:04:11Z" }
TTL: ~60s, refreshed by connection heartbeat
-- routing table for delivery; expiry self-heals crashed gateways
- Send dedupe (retry safety)
Key: "dedupe:{sender_id}:{client_msg_id}" -> message_id
TTL: 24h
-- a retried send returns the original message_id instead of a twin
- Typing indicators (never persisted anywhere)
Key: "typing:{conversation_id}" -> set of user_ids, TTL a few seconds{
"type": "send",
"client_msg_id": "c_a1b2c3",
"conversation_id": "conv_42",
"msg_type": "text",
"ciphertext": "base64-encrypted-on-the-phone..."
}{
"type": "ack",
"client_msg_id": "c_a1b2c3",
"message_id": "m_9f3e7d",
"seq": 1248,
"status": "sent"
}
// client_msg_id is the sender's retry handle: if this ack is lost and the
// phone resends, the server recognizes the id and re-acks with the SAME
// message_id and seq - no duplicate is ever created. The server stores
// only ciphertext; it cannot read what it is delivering.{
"type": "receipt",
"conversation_id": "conv_42",
"seq": 1248,
"status": "delivered"
}{
"type": "message",
"conversation_id": "conv_42",
"message_id": "m_9f3e7d",
"seq": 1248,
"sender_id": "u_A",
"msg_type": "text",
"ciphertext": "base64...",
"sent_at": "2026-07-05T18:04:11Z"
}
// The receiving phone checks seq against what it already has: a gap
// (it holds 1247 but receives 1250) triggers a range fetch for the
// missing messages; a repeat of 1248 is silently dropped. The receipt
// it sends back becomes the sender's second check mark.{
"conversation_id": "conv_42",
"content_hash": "sha256:7d865e959b2466918c98...",
"size_bytes": 2048576,
"content_type": "image/jpeg"
}{
"already_stored": false,
"upload_url": "https://blobs.example.com/media/...(presigned)",
"media_pointer": "blob_4c8e6f2a",
"expires_at": "2026-08-04T18:04:11Z"
}
// The blob is encrypted on the phone BEFORE upload; the decryption key
// travels inside the (end-to-end encrypted) message, never to the server.
// If content_hash is already stored - a popular forward - already_stored
// is true and no bytes are uploaded at all: the Design Dropbox dedup
// pattern applied to viral media.{
"type": "group",
"name": "Trip Planning",
"member_ids": ["u_A", "u_B", "u_C"]
}{
"conversation_id": "conv_88",
"type": "group",
"name": "Trip Planning",
"members": [
{ "user_id": "u_A", "role": "admin" },
{ "user_id": "u_B", "role": "member" },
{ "user_id": "u_C", "role": "member" }
],
"created_at": "2026-07-05T18:04:11Z"
}{
"type": "backlog",
"messages": [
{ "conversation_id": "conv_42", "seq": 1249, "sender_id": "u_A",
"msg_type": "text", "ciphertext": "..." },
{ "conversation_id": "conv_42", "seq": 1250, "sender_id": "u_A",
"msg_type": "media", "media_pointer": "blob_4c8e6f2a",
"ciphertext": "..." },
{ "conversation_id": "conv_88", "seq": 12, "sender_id": "u_C",
"msg_type": "text", "ciphertext": "..." }
],
"has_more": false
}
// The inbox drains in order per conversation; the phone acks in batches.
// Each ack deletes an inbox pointer and fires a "delivered" receipt to
// the sender - which is why landing a plane produces a burst of two-check
// notifications on other people's phones.Sequence number: a per-conversation counter (1247, 1248, 1249) assigned by the conversation's single owning shard. It gives every message an unambiguous position, which is what makes gap detection ("I have 1248, I just got 1250, fetch 1249") and duplicate detection ("I already have 1248, drop it") one integer comparison each. Clocks cannot do this job - counters from one referee can. At-least-once: the delivery promise a network can actually keep. Every hop retries until it hears an acknowledgment, which guarantees the message arrives - possibly more than once, when an ack (rather than the message) is what got lost. Idempotent: an operation that is safe to repeat - doing it twice equals doing it once. Receiving is made idempotent by ids: the server dedupes retried sends by client_msg_id, phones dedupe re-pushed messages by seq. At-least-once delivery plus idempotent receipt at every hop is how the system behaves exactly-once without any hop actually promising it - the same layered honesty as retried chunk uploads in Design Dropbox.