Design WhatsApp: System Design & Architecture

Build a messaging platform where a message typed on one phone shows up on another phone anywhere in the world within a fraction of a second - reliably, in order, end-to-end encrypted, and even when the recipient is offline.

Scope

In Scope

  • One-on-one and group messaging (groups capped at ~1,000 members) with real-time delivery
  • Message states and receipts: sent, delivered, read - the one and two check marks
  • Reliable delivery with per-conversation ordering: messages never arrive out of order or get silently lost
  • Offline delivery: messages queue for a disconnected recipient and arrive when they come back
  • Media messages (photos, videos, documents) via blob storage and CDN
  • End-to-end encryption as a design constraint (the server relays ciphertext it cannot read)

Out of Scope

  • Voice and video calling (real-time media streaming is a different problem with different infrastructure)
  • The cryptographic protocol internals (Signal Protocol key ratchets): we treat encryption as a black box and design around its consequences
  • Status/stories and broadcast channels (feed-shaped features - see Design a Newsfeed)
  • Message history backup and multi-device history sync (device-to-device transfer of encrypted archives is its own design)
  • Phone number verification, OTP delivery, and carrier integration
  • Business messaging APIs, payments, and spam/abuse detection internals

💡 What does end-to-end encryption change about the design?

End-to-end encryption (E2EE) means messages are encrypted on the sender's phone and decrypted only on the recipient's phone. The server passes along ciphertext - scrambled bytes it has no key for. That one constraint quietly reshapes the whole system. The server cannot search message content, cannot scan it for moderation, and cannot show your chat history on a new phone, because it never sees any content in readable form. It also means the server has little reason to keep messages at all: once a message is delivered to every recipient, the server deletes its copy. The server is a relay, not an archive - your chat history lives on your devices, not in our database. That is the opposite of most systems in this practice set, and it is why the storage estimates below come out surprisingly small. What the server still sees is metadata: who messaged whom, when, and how big the message was. It needs some of that to route and deliver. Treating metadata as sensitive (minimal retention, minimal logging) is part of taking E2EE seriously, and worth saying in an interview.

Functional Requirements

  • A user can send a text message to another user, and it arrives in real time if they are online
  • A user can create a group, add or remove members, and message everyone in it
  • A sender sees the state of each message: sent (server has it), delivered (their phone has it), read (they saw it)
  • Messages in a conversation appear in the same order for everyone, with nothing missing and nothing duplicated
  • A user who was offline receives everything they missed, in order, when they reconnect
  • A user can send photos, videos, and documents
  • A user can see basic presence: whether a contact is online and when they were last seen

Non-functional Requirements

  • Delivery latency: p95 < 500ms sender-to-recipient within a region when both are online
  • Scale: 2B monthly users, ~1B daily actives, ~100B messages/day, peak ~200M devices connected at once
  • Reliability: an acknowledged message is never lost - it is durably stored until every recipient has it
  • Ordering: strict, gap-free ordering within each conversation (no global ordering promise across conversations)
  • Offline retention: undelivered messages are kept for 30 days, then dropped
  • Availability: 99.99% for the messaging path; presence and typing indicators are best-effort and may degrade first

💡 What exactly do the check marks promise, and what does "ordering" mean here?

Each check mark is an acknowledgment traveling back along the path the message took. One check (sent) means the server has durably stored the message - if your phone dies right now, the message still exists. Two checks (delivered) means the recipient's phone has it - the server's job is done and it can delete its copy. Blue checks (read) mean the recipient actually opened the conversation. Each is a tiny message flowing backwards, and they ride the exact same delivery pipeline as normal messages. Ordering is promised per conversation, not globally. Within one chat, everyone sees messages in the same order with no gaps: message 1248 always comes after 1247. Across different chats there is no promise at all - a message in your family group and one from a coworker have no defined order between them, and nobody would ever notice. That narrow promise is a gift to the architecture: because ordering only matters within a conversation, we can shard the system by conversation and let each shard order its own messages independently, with no coordination between shards. A global ordering guarantee would force every message in the world through some agreement step - vastly more expensive, for a property no user can even observe.

Back-of-the-Envelope Estimates

  • Assume 2B monthly active users, ~1B daily actives, peak ~200M devices holding an open connection at once
  • A write is a message send (or a receipt/state change); a read is a device receiving messages - mostly pushed over its connection rather than requested
  • Message volume: ~100B messages/day ÷ 86,400 ≈ 1.2M messages/sec average; spikes (New Year's midnight) run 3-5x → design for ~5M/sec bursts
  • Deliveries exceed sends: with group chats, the average message reaches ~2-3 devices → ~3M deliveries/sec average. Groups are capped at ~1,000 members, so the worst single message costs ~1,000 deliveries - bounded by product design
  • Connections: 200M concurrent ÷ ~65K connections per gateway server ≈ ~3,000 gateway servers - the connection fleet is a first-class cost here, unlike request/response systems
  • Text storage is tiny because the server deletes messages once delivered: only in-flight messages are stored. If ~10% of a day's messages wait for an offline recipient for ~12 hours, that is ~10B messages x ~1KB ≈ 10TB rolling - a rounding error
  • Media dominates storage: if ~1 in 20 messages carries media at ~400KB average, that is ~5B files/day ≈ 2PB/day uploaded. With a 30-day server retention (delivered media cached for re-download, then dropped), the rolling footprint is tens of PB in blob storage - the one genuinely big number in the system

💡 What the estimates mean: this is a routing problem, not a storage problem

The numbers say something unusual: 1.2M writes/sec sounds like a storage nightmare, but because the server deletes every message once it is delivered, almost nothing accumulates. The hard problem is not where to keep 100B messages - it is how to move 100B messages/day to the right phones, in order, without losing any. The design should be judged as a delivery pipeline, not a database. What storage remains splits three ways. In-flight messages (sent but not yet delivered to everyone) go to Cassandra, partitioned by conversation_id and ordered by a sequence number within each partition: appends are cheap, and "give me everything in this chat after sequence N" is a single ordered range read - exactly the query an offline device asks when it reconnects. This is the same append-and-read-forward shape as the change feed in Design Dropbox. Ephemeral state - who is online, which gateway holds each device's connection, typing indicators - lives in Redis with TTLs. It is rewritten constantly and losing it costs nothing but a moment of staleness, the same "disposable data gets the cheap path" argument as presence in Design Excalidraw. Small relational truth - accounts, group membership, public key registry - lives in PostgreSQL. It is modest in size, changes rarely, and wants constraints (one account per phone number). Media takes the Dropbox route entirely: encrypted blobs go straight to object storage via presigned URLs, downloads come from CDN edges, and the message itself carries only a pointer and a decryption key. The 2PB/day of media never touches the messaging pipeline. The contrast worth naming: Design a Newsfeed stores content once and serves it to readers forever; WhatsApp delivers each message once to a known small set of devices and then forgets it. Store-and-serve versus route-and-delete - they lead to completely different systems.

High-level Architecture

  • Gateway fleet (WebSocket): ~3,000 servers holding ~200M persistent connections; each device connects to one gateway, which pushes messages down the open socket the moment they arrive
  • Session Registry (Redis): the routing table - user_id → which gateway holds their connection right now, plus presence (online / last_seen). Updated on connect/disconnect, checked on every delivery
  • Chat Service (sharded by conversation_id): the referee for each conversation - assigns the per-conversation sequence number, durably stores the message, and hands it to delivery. One owner per conversation is what makes gap-free ordering cheap
  • Message Store (Cassandra, partitioned by conversation): in-flight messages only, ordered by sequence number, deleted once every recipient has acknowledged (30-day TTL as the backstop)
  • Inbox (per-recipient queue): pointers to undelivered messages per user, so a reconnecting device drains one queue instead of asking every conversation "did I miss anything?"
  • Group Service (PostgreSQL): group metadata and membership - the list the fan-out consults when a group message needs delivering to ~1,000 inboxes
  • Media Service + Blob Storage + CDN: presigned uploads of client-encrypted blobs, CDN downloads, content-hash deduplication for popular forwards (the Design Dropbox pattern end to end)
  • Push Notification hook (APNs/FCM): for fully offline devices, a wake-up ping - the content stays encrypted and arrives through the normal pipeline when the app opens

Data/User Flow Diagram

SEND - one-on-one message from A to B
[A's phone] ==WebSocket==> [Gateway 12]
                               |
                               v
                    [Chat Service - owner of conv_42]
                    assign seq 1248 -> store in Message Store
                               |
                    ack back to A  ("sent" - one check)
                               |
                               v
                    [Session Registry (Redis)] -- where is B?
                          |                |
                     B online          B offline
                          |                |
                          v                v
                 [Gateway 87]        [Inbox (B's queue)]
                  ==push==> B             +
                          |          [Push notification "wake up"]
              B's phone acks              |
                          |          B reconnects later ->
                          v          drain inbox in order -> ack each
              "delivered" (two checks) flows back to A
              Message Store deletes the delivered message

GROUP - one message, many inboxes
[Chat Service] -> [Group Service] members? -> write pointer to each
                  member's Inbox -> push to the online ones via their
                  gateways (bounded: groups cap at ~1,000 members)

MEDIA - bytes never touch the messaging pipeline
[A] --presigned upload--> [Blob Storage (client-encrypted)] --> [CDN]
     the message carries only a pointer + decryption key; B downloads
     from the CDN edge (same direct-transfer pattern as Design Dropbox)

💡 How does the system find the one phone a message should go to?

With 200M open connections spread over ~3,000 gateways, delivery starts with a lookup problem: which gateway is holding the recipient's socket right now? The Session Registry answers it. When a device connects, its gateway writes "user u_B is on gateway 87" into Redis; when the socket drops, the entry is cleared (with a TTL as the safety net for crashed gateways). Delivering a message means one registry read, then a hop to that gateway, which pushes the message down the open socket. Compare this with Design Excalidraw, which routes sticky by board so that everyone editing a board lands on the same server and shares its memory. Chat cannot do that - you and your contact should not need to share a server, because you each talk to many people. So instead of co-locating conversation participants, WhatsApp keeps a directory of where everyone is and routes hop-by-hop: registry lookup, gateway-to-gateway hop, socket push. Sticky routing moves people to the state; a session registry moves messages to the people. Knowing which one fits which problem is a genuinely useful interview instinct.

Data Model & Storage

  • Users (PostgreSQL): user_id, phone_number (unique), display_name, device tokens for push, registered public keys per device, last_seen policy. Small, relational, constraint-friendly - one account per phone number is a database rule, not an application hope.
  • Conversations + Group Members (PostgreSQL): conversation_id, type (direct/group), name, and a membership table (conversation_id, user_id, role). Membership is the list fan-out consults, and an index on user_id answers "which conversations am I in?"
  • Messages (Cassandra, partitioned by conversation_id, clustered by seq): the in-flight store. The partition key makes each conversation one ordered strip of data; the clustering key (seq) makes "everything after N" a cheap range read. Rows hold ciphertext the server cannot read, and are deleted on full delivery (TTL 30 days as backstop).
  • Inbox (Cassandra, partitioned by user_id): per-recipient pointers (conversation_id, seq) to messages awaiting delivery - the same pointers-not-copies move as the Feed Store in Design a Newsfeed. A reconnecting device drains its one inbox instead of polling every conversation.
  • Presence + Session Registry (Redis, TTL): user_id → gateway, online flag, last_seen. Rewritten constantly, worthless if stale by a minute, perfect for an in-memory store with expiry - the Design Excalidraw presence argument verbatim.
  • Dedupe keys (Redis, TTL ~1 day): sender's client_msg_id → assigned message_id, so a retried send returns the original message instead of creating a twin.
  • Media (Blob Storage + CDN): client-encrypted blobs, content-hash addressed so a viral forward is stored once, expiring after the retention window.

💡 Why is the message database almost empty in a system moving 100B messages a day?

Because delivery is the product, not storage. A message exists on the server only for the gap between "sender's phone sent it" and "every recipient's phone has it" - typically milliseconds, at most 30 days. Two forces make this the right call. First, E2EE makes stored messages nearly useless to us: we cannot search them, index them, or render them on a new device - they are opaque ciphertext. Keeping them would be all liability (a growing pile of intercepted-looking data) and no feature. Second, deleting-on-delivery turns the scaling problem from "store 36 trillion messages a year" into "hold roughly one hour's worth of undelivered traffic," which fits in a few terabytes. The Message Store behaves like a buffer, not a library. The Inbox follows the Newsfeed pointer trick: entries are (conversation, seq) references, ~50 bytes each, never copies of content. One message to a 1,000-member group writes one message row and 1,000 tiny pointers - and when each member's phone acknowledges, its pointer is deleted, the ref-count drops, and eventually the message row itself goes. Small, self-cleaning structures everywhere.

API Design

WS /v1/gateway
The persistent connection. Sends, deliveries, acks, receipts, presence, and typing all flow over this one socket.
POST /v1/conversations
Create a direct conversation or a group (name + initial members).
PUT /v1/conversations/:id/members
Add or remove group members (admin only).
POST /v1/media/upload-init
Get a presigned URL to upload a client-encrypted media blob; returns the pointer to embed in a message.
GET /v1/users/:id/keys
Fetch a contact's public key bundle so a new device or contact can encrypt to them.
GET /v1/conversations
List conversation metadata (names, members, own last-read markers). Note what is absent: no message history endpoint - the server does not have it.

Low-level Design

Chat Service - sending a message (the write path, step by step):

  • 1) A's phone sends over its socket: a client-generated client_msg_id (its retry/dedupe handle), the conversation_id, and the ciphertext.
  • 2) The gateway forwards to the Chat Service shard that owns conv_42. The owner assigns the next sequence number (1248), writes the message durably to the Message Store, and records client_msg_id → message_id in the dedupe cache.
  • 3) The owner acks the sender - A's phone shows one check. From this moment the message survives anything, including A's phone dying.
  • 4) Delivery: look up B in the Session Registry. Online → hop to B's gateway, push down the socket. Offline → write a pointer to B's inbox and fire a push notification wake-up.
  • 5) B's phone acknowledges receipt; the "delivered" receipt travels back to A (two checks), B's inbox pointer is removed, and once all recipients have acked, the Message Store deletes the row.
  • Every arrow in this flow can fail and be retried, which is why every step is idempotent - see the delivery-guarantees flow.

Chat Service - ordering (the genuinely tricky part):

  • The promise: everyone in a conversation sees the same messages in the same order, gap-free.
  • The mechanism: each conversation has exactly one owning shard, and that owner assigns a per-conversation counter - 1247, 1248, 1249. One referee per conversation means there is never a tie to break, the same single-owner move as the board-owning gateway in Design Excalidraw.
  • Why not timestamps: phone clocks are wrong by seconds to minutes, and even server clocks skew across machines - two messages in the same millisecond have no defined order. A counter from a single owner is trivially total, and costs one increment.
  • Receivers use the sequence to self-heal: if a phone holding seq 1248 receives 1250, it knows 1249 is missing and asks the store for the gap (a range read) before rendering; if it receives 1248 twice, it drops the duplicate.
  • Scope of the promise: per conversation only. No cross-conversation ordering means no cross-shard coordination - that narrowness is what lets the whole thing scale by adding shards.

Delivery guarantees - at-least-once plus idempotent receive:

  • Nothing on a network can promise exactly-once delivery of an attempt; what you can build is at-least-once with deduplication, which behaves like exactly-once.
  • Sender → server: A's phone resends until it gets the ack. If the ack (not the message) was what got lost, the server sees the same client_msg_id, does not create a twin, and re-acks with the original message_id and seq.
  • Server → recipient: the server re-pushes until B's phone acks. If B got it but the ack was lost, B receives seq 1248 a second time and silently drops it.
  • The pattern at every hop is identical: retry on silence, dedupe by id on arrival.
  • Say the summary line in the interview: reliable messaging is at-least-once delivery plus idempotent receipt at every hop - "exactly-once" is the emergent behavior, not a network primitive.

Sync - coming back online after a dead zone (step by step):

  • 1) The phone reconnects to any gateway, authenticates, and the gateway writes its location into the Session Registry - the user is routable again.
  • 2) The phone drains its inbox: pointers come out in order, the referenced messages are fetched from the Message Store, pushed down the socket, and acked in batches.
  • 3) Each ack deletes the inbox pointer and fires the delivered receipt to the original sender - which is why a plane-load of two-check notifications hits senders the moment someone lands and reconnects.
  • 4) A device offline past 30 days finds an empty (expired) inbox: those messages are gone, by policy, and the sender's single check is the honest record that delivery never happened.
  • The inbox is a per-user cursor into pending work - the same read-forward-from-a-bookmark shape as the change feed in Design Dropbox.

Group Service - fan-out with a bounded blast radius:

  • A group message is stored once (one row in the conversation's partition), then fanned out as pointers: one inbox write per member, push to the online ones.
  • The cap is the design: groups max out around ~1,000 members, so the worst message costs ~1,000 pointer writes - always affordable, no special path needed. Contrast with Design a Newsfeed, where follower counts are unbounded and a 20M-follower account forces a switch to fan-out-on-read. Chat chose to cap the product instead of engineering for the uncapped case - a product decision doing the work of an architecture.
  • Receipts aggregate: the sender sees "delivered" when all members' devices have acked (a counter on the message, not 1,000 separate notifications).
  • Membership changes take effect at the sequence number where they land: someone added at seq 500 gets 501 onward, and none of the encrypted history before - which E2EE would make undecryptable for them anyway.

Media Service - photos and videos without touching the pipeline:

  • 1) A's phone encrypts the file, then uploads it straight to blob storage via a presigned URL (the exact Design Dropbox direct-upload flow).
  • 2) The message sent through the normal pipeline is tiny: a pointer to the blob plus the decryption key - the 2PB/day of media never flows through gateways or the Chat Service.
  • 3) Recipients download from the CDN edge and decrypt locally.
  • 4) Forwards exploit content-hash addressing: a viral video forwarded a million times is stored once, every copy of the message pointing at the same blob - block-level dedup's simpler cousin.
  • 5) Blobs expire after the retention window; phones that already downloaded keep their local copies, honoring relay-not-archive.

Database Types and Schemas

Database Schemas

- Users (PostgreSQL)
  user_id UUID PRIMARY KEY,
  phone_number VARCHAR(20) UNIQUE NOT NULL,  -- one account per number: a DB rule
  display_name VARCHAR(100),
  device_tokens JSONB,           -- push notification handles per device
  public_keys JSONB,             -- registered per-device public key bundles
  last_seen_policy policy_enum DEFAULT 'contacts',
  created_at TIMESTAMP DEFAULT NOW()

- Conversations (PostgreSQL)
  conversation_id UUID PRIMARY KEY,
  type conv_type_enum NOT NULL,          -- 'direct' | 'group'
  name VARCHAR(100),                     -- groups only
  created_by UUID REFERENCES users(user_id),
  created_at TIMESTAMP DEFAULT NOW()

- Group Members (PostgreSQL)
  conversation_id UUID NOT NULL,
  user_id UUID NOT NULL,
  role member_role_enum DEFAULT 'member',  -- 'member' | 'admin'
  joined_at_seq BIGINT,   -- membership starts at this sequence number
  PRIMARY KEY (conversation_id, user_id)
  INDEX idx_member_convs ON (user_id)      -- "which chats am I in?"

- Messages (Cassandra) -- IN-FLIGHT ONLY: deleted once fully delivered
  PRIMARY KEY ((conversation_id), seq)
  Fields: message_id UUID, sender_id, ciphertext BLOB,
          msg_type ('text'|'media'|'system'), media_pointer, sent_at,
          recipients_pending INT   -- delivered-to-all when this hits 0
  TTL: 30 days (the offline-retention backstop)
  -- partition per conversation = one ordered strip of data
  -- clustering by seq makes "everything after N" a single range read
  -- (the old design keyed this by message_id, which cannot answer
  --  either ordering or catch-up queries - the partition key IS the design)

- Inbox (Cassandra) -- per-recipient pointers to undelivered messages
  PRIMARY KEY ((user_id), conversation_id, seq)
  Fields: enqueued_at
  TTL: 30 days
  -- ~50-byte pointers, never message copies (the Newsfeed pointer move)
  -- drained in order on reconnect; each ack deletes its row

Redis keys

- Session Registry + Presence
  Key: "sess:{user_id}" -> { "gateway": "gw-87", "online": true,
                             "last_seen": "2026-07-05T18:04:11Z" }
  TTL: ~60s, refreshed by connection heartbeat
  -- routing table for delivery; expiry self-heals crashed gateways

- Send dedupe (retry safety)
  Key: "dedupe:{sender_id}:{client_msg_id}" -> message_id
  TTL: 24h
  -- a retried send returns the original message_id instead of a twin

- Typing indicators (never persisted anywhere)
  Key: "typing:{conversation_id}" -> set of user_ids, TTL a few seconds

API Request/Response Payloads

WS send: a message (and its ack)
Request
{
  "type": "send",
  "client_msg_id": "c_a1b2c3",
  "conversation_id": "conv_42",
  "msg_type": "text",
  "ciphertext": "base64-encrypted-on-the-phone..."
}
Response
{
  "type": "ack",
  "client_msg_id": "c_a1b2c3",
  "message_id": "m_9f3e7d",
  "seq": 1248,
  "status": "sent"
}

// client_msg_id is the sender's retry handle: if this ack is lost and the
// phone resends, the server recognizes the id and re-acks with the SAME
// message_id and seq - no duplicate is ever created. The server stores
// only ciphertext; it cannot read what it is delivering.
WS receive: an incoming message (and the receipt back)
Request
{
  "type": "receipt",
  "conversation_id": "conv_42",
  "seq": 1248,
  "status": "delivered"
}
Response
{
  "type": "message",
  "conversation_id": "conv_42",
  "message_id": "m_9f3e7d",
  "seq": 1248,
  "sender_id": "u_A",
  "msg_type": "text",
  "ciphertext": "base64...",
  "sent_at": "2026-07-05T18:04:11Z"
}

// The receiving phone checks seq against what it already has: a gap
// (it holds 1247 but receives 1250) triggers a range fetch for the
// missing messages; a repeat of 1248 is silently dropped. The receipt
// it sends back becomes the sender's second check mark.
POST /v1/media/upload-init
Request
{
  "conversation_id": "conv_42",
  "content_hash": "sha256:7d865e959b2466918c98...",
  "size_bytes": 2048576,
  "content_type": "image/jpeg"
}
Response
{
  "already_stored": false,
  "upload_url": "https://blobs.example.com/media/...(presigned)",
  "media_pointer": "blob_4c8e6f2a",
  "expires_at": "2026-08-04T18:04:11Z"
}

// The blob is encrypted on the phone BEFORE upload; the decryption key
// travels inside the (end-to-end encrypted) message, never to the server.
// If content_hash is already stored - a popular forward - already_stored
// is true and no bytes are uploaded at all: the Design Dropbox dedup
// pattern applied to viral media.
POST /v1/conversations
Request
{
  "type": "group",
  "name": "Trip Planning",
  "member_ids": ["u_A", "u_B", "u_C"]
}
Response
{
  "conversation_id": "conv_88",
  "type": "group",
  "name": "Trip Planning",
  "members": [
    { "user_id": "u_A", "role": "admin" },
    { "user_id": "u_B", "role": "member" },
    { "user_id": "u_C", "role": "member" }
  ],
  "created_at": "2026-07-05T18:04:11Z"
}
WS backlog: reconnecting after being offline
Response
{
  "type": "backlog",
  "messages": [
    { "conversation_id": "conv_42", "seq": 1249, "sender_id": "u_A",
      "msg_type": "text", "ciphertext": "..." },
    { "conversation_id": "conv_42", "seq": 1250, "sender_id": "u_A",
      "msg_type": "media", "media_pointer": "blob_4c8e6f2a",
      "ciphertext": "..." },
    { "conversation_id": "conv_88", "seq": 12, "sender_id": "u_C",
      "msg_type": "text", "ciphertext": "..." }
  ],
  "has_more": false
}

// The inbox drains in order per conversation; the phone acks in batches.
// Each ack deletes an inbox pointer and fires a "delivered" receipt to
// the sender - which is why landing a plane produces a burst of two-check
// notifications on other people's phones.

💡 Three terms from the flows: sequence number, at-least-once, idempotent

Sequence number: a per-conversation counter (1247, 1248, 1249) assigned by the conversation's single owning shard. It gives every message an unambiguous position, which is what makes gap detection ("I have 1248, I just got 1250, fetch 1249") and duplicate detection ("I already have 1248, drop it") one integer comparison each. Clocks cannot do this job - counters from one referee can. At-least-once: the delivery promise a network can actually keep. Every hop retries until it hears an acknowledgment, which guarantees the message arrives - possibly more than once, when an ack (rather than the message) is what got lost. Idempotent: an operation that is safe to repeat - doing it twice equals doing it once. Receiving is made idempotent by ids: the server dedupes retried sends by client_msg_id, phones dedupe re-pushed messages by seq. At-least-once delivery plus idempotent receipt at every hop is how the system behaves exactly-once without any hop actually promising it - the same layered honesty as retried chunk uploads in Design Dropbox.

Concluding Discussion

Potential Bottlenecks

  • Issue: Synchronized global spikes - New Year's midnight pushes sends to 3-5x normal in a single minute.
    Mitigation: gateways and Chat Service shards scale horizontally, in-flight storage buffers the surge, and best-effort traffic (presence, typing, read receipts) is shed first so raw message delivery keeps its capacity - degrade the disposable, protect the promise.
  • Issue: A hyperactive 1,000-member group concentrates traffic on one conversation partition - every message is one hot-partition write plus 1,000 inbox writes.
    Mitigation: the group cap bounds the worst case by design; fan-out workers batch inbox writes; per-conversation rate limits stop a runaway bot from monopolizing a shard.
  • Issue: 200M idle connections cost memory and heartbeat chatter before any message is sent.
    Mitigation: tune per-connection footprint (this is why chat companies obsess over connections-per-server), lengthen heartbeat intervals for backgrounded devices, and lean on push notifications instead of sockets for the long-idle tail.

Single Points of Failure

  • The Session Registry: if routing is down, real-time push stalls. Run Redis clustered and replicated - and note the graceful floor: delivery falls back to inbox + push notification, so messages are delayed, never lost.
  • A Chat Service shard owning a conversation: its crash pauses that conversation's ordering. A replacement owner reads the highest stored seq and resumes counting; senders' unacked messages retry automatically, and dedupe ids make the retries safe.
  • Apple/Google push services: an outage outside our control. Circuit-break and queue - online delivery over sockets is unaffected, and offline users receive everything from their inbox on next app open, just without the wake-up tap.

Security Improvements

  • E2EE handles content; the remaining crown jewel is metadata (who talks to whom, when). Minimize retention and logging of it - deleted-on-delivery is a privacy feature as much as a storage one.
  • Key-change safety: if a contact's key suddenly changes (SIM swap, re-registration, or an attacker), surface it in the conversation and let users verify safety numbers - the defense against silent man-in-the-middle re-registration.
  • Rate-limit sends per user and per conversation at the gateway; spam detection must work on metadata and reports alone, since content is unreadable by design.
  • Media URLs are presigned and expiring, and blobs are client-encrypted - a leaked link yields ciphertext, and only briefly.

Critical Knowledge

If the interviewer asks "why WebSockets instead of polling?":

  • Same core answer as Design Excalidraw: chat needs the server to push the instant something happens; polling from a billion phones would be almost entirely wasted requests asking "anything new?"
  • The chat-specific twist is the scale of the connection fleet: ~200M concurrent sockets across ~3,000 gateways, which makes connections-per-server a first-class engineering metric and adds the routing problem - the Session Registry exists because holding the socket and knowing who holds the socket are two different jobs.
  • Mention the battery angle for mobile: one quiet socket plus OS push notifications beats any polling interval on power.
  • Cost: long-lived connections pin memory and demand graceful drain on deploys - the operational price of push.

If the interviewer asks "how do you guarantee message order?":

  • Scope the promise first: order within a conversation, no promise across conversations - users cannot observe cross-chat order, and dropping that requirement removes all cross-shard coordination.
  • Mechanism: one owning shard per conversation assigns a monotonic sequence number - a single referee, so no distributed tie-breaking, the same move as Excalidraw's board owner.
  • Kill the naive answer: timestamps cannot do this - device clocks are minutes wrong, server clocks skew, same-millisecond messages have no order.
  • Close the loop on the receiver: sequence numbers let phones detect gaps (fetch the range) and duplicates (drop), so ordering is verified at the edge, not just asserted centrally.

If the interviewer asks "what happens if the recipient's phone is off?":

  • The message is already safe - one check means durably stored, sender's job done. A pointer sits in the recipient's inbox and a push notification tries to wake the device.
  • On reconnect the phone drains its inbox in order, acks in batches, and every ack fires a delivered receipt back to the sender - the burst of check marks when someone lands from a flight.
  • After 30 days undelivered, inbox and message expire: the sender's stuck-at-one-check is the honest record that delivery never happened.
  • Underline the philosophy: the server is a relay with a 30-day buffer, not an archive - E2EE makes stored ciphertext useless to the server anyway, so delete-on-delivery costs no feature and buys privacy and tiny storage.

If the interviewer asks "how is a message never lost and never duplicated?":

  • Be precise: no network hop can promise exactly-once; you build at-least-once plus idempotent receive, and exactly-once is the emergent behavior.
  • Never lost: every hop retries until acked - phone resends to server, server re-pushes to phone, and the message stays stored until every recipient acks.
  • Never duplicated: every retry carries the same identity - client_msg_id dedupes sender retries at the server, seq dedupes server re-pushes at the phone - so repeats collapse into no-ops.
  • Name the invariant: retry on silence, dedupe by id on arrival, at every hop. That one sentence transfers to payments, queues, and webhooks - interviewers know it generalizes.

If the interviewer asks "how is group fan-out different from newsfeed fan-out?":

  • Same verb, opposite constraint. A group message is stored once and fanned out as ~1,000 inbox pointers at worst, because membership is capped - the expensive case is bounded by product design, so plain push-to-all always works.
  • Design a Newsfeed has no cap: a 20M-follower account makes fan-out-on-write ruinous, forcing the hybrid push/pull with a hot-user threshold.
  • The transferable insight: a product constraint (max group size) can do the work of an architecture (celebrity pull paths) - cheaper to build, cheaper to run, and worth proposing in interviews before reaching for machinery.
  • Receipts need the same discipline: aggregate "delivered to all" as a counter, or a 1,000-member group turns every message into a thousand receipt notifications.