Build a real-time collaborative document editor where many people can type into the same document at once, see each other's edits and cursors within milliseconds, and never lose work. The hard part is not storing files (that is the Dropbox problem); it is merging everyone's keystrokes into one consistent document without conflicts.
Dropbox stores whole files and syncs them across your devices. It deliberately skipped real-time co-editing: there, "saving" means uploading a new version of the whole file now and then, and a conflict (two devices editing offline) is resolved by just keeping both copies as "file (conflicted copy)". Google Docs is the opposite. The unit of change is not a whole file, it is a single tiny edit like "insert the letter h at position 12". Many people apply these tiny edits to the same document at the same moment, and the system must merge them into one shared document that everyone sees identically, in real time. You cannot solve that by keeping two copies, because there is only one document and everyone is looking at it live. That merging problem is the entire reason this question exists, so most of our design is about it.
Convergence means that after all the in-flight edits have been delivered, every client shows byte-for-byte the same document. We do NOT require that all clients are identical at every single instant, and here is why: you and I are in different cities, so there is unavoidable network delay between us. For a few hundred milliseconds my screen may show my latest keystroke before yours has arrived, and vice versa. That brief difference is fine and unavoidable. What we DO require is that once the edits finish flowing, we both land on the same final text, and neither of us ever sees garbled or half-merged text in the meantime. That is the "strong convergence" guarantee, and it is the whole point of the merging algorithm. On the CAP theorem: a single document lives behind one coordinator (explained later) that puts all edits for that document into one order, so per-document we lean toward consistency and accept that if that coordinator is briefly unreachable, editing that one document pauses for a moment while a replacement takes over. We are NOT trying to keep editing usable during a partition at the cost of two diverging documents, because two diverging copies of a live shared document is exactly the failure we cannot tolerate.
14M tiny operations per second is the number that shapes everything. Each op is only ~100-200 bytes, so total bandwidth is modest (a few GB/s), but the operation RATE is enormous and every op has to be ordered correctly relative to other ops on the same document. That tells us two things. First, this is NOT a job for a CDN or blob storage. Dropbox could push huge files to a CDN because files are big, immutable, and read far more than written. Here the data is the opposite: a stream of tiny mutable edits that must be merged in order. CDNs cache static bytes; they cannot merge concurrent keystrokes. So the live path is application servers holding open connections, not edge caches. Second, the work naturally shards by document. All edits to one document must be ordered together, but edits to two different documents are completely independent. So we route every connection for a given document to one coordinator process that owns that document, and we spread the 10M active documents across many servers. One hot document with 100 editors is still only ~700 ops/sec, which one process handles easily; the scale challenge is the sheer NUMBER of documents, which we solve by horizontal sharding, not by making any single document faster. Third, durability has to keep up with the write rate cheaply. We cannot do a full database transaction per keystroke at 14M/sec. So we append edits to a durable op log (an append-only log, very cheap to append to) and only periodically fold them into a compact snapshot. That append-then-snapshot pattern is what makes continuous autosave affordable.
CLIENT A (browser tab) CLIENT B (browser tab)
| ^ | ^
my edits | | others' merged edits | |
(ops) v | (pushed live) v |
[ WebSocket Gateway ] <------> [ WebSocket Gateway ]
| |
+-------------+----------------+
| (all connections for doc X
v routed to its one owner)
[ Document Coordinator for doc X ]
- assigns each op an order
- merges concurrent ops (OT)
- broadcasts merged op to all clients
|
+-------------+--------------------+
v v
[ Op Log (durable, [ Presence (Redis, in-memory)
append-only, per doc) ] cursors + who is online ]
|
| (periodic compaction)
v
[ Snapshot Store ] <--- late joiner loads snapshot + recent ops
[ Metadata DB ] <--- title, owner, permissions, version
Open doc: client -> Gateway -> Coordinator -> load latest snapshot + ops since it -> send full state -> open live stream
Type: client -> sends op (with its base version) -> Coordinator orders + merges (OT) -> appends to Op Log -> broadcasts merged op to everyone
Reconnect: client -> "I'm at version N" -> Coordinator sends every op after N -> client replays them, then resumes liveWebSocket: a normal web page asks the server for things one request at a time, so the server can only answer when asked. That is fine for loading a page, but useless for live editing, because the server needs to PUSH your collaborator's keystroke to you the instant it happens, without you asking. A WebSocket is a connection that stays open in both directions: you send your edits up it, and the server sends everyone else's edits down it, with no repeated "anything new?" polling. This is why we use WebSockets instead of the periodic polling Dropbox used for sync; polling every 30 seconds would make co-editing feel broken. One coordinator owns the document: the reason all edits for a document go through a single process is that merging concurrent edits only works if someone decides the order they happened in. If two servers both accepted edits to the same document independently, they could order them differently and the two halves of the document would drift apart. So we make one process the single owner (the coordinator) for each document; it is the one place that decides "this edit came before that one." Different documents have different owners, so the system still scales out across millions of documents.
The op log and presence are treated completely differently on purpose, because losing them has completely different costs. The op log is the document. Every accepted edit is appended to it before we tell the client "saved", so even if the coordinator crashes one millisecond later, the edit survives and the document can be rebuilt by replaying the log. This is the autosave: there is no "Save" button and no periodic whole-document upload. Each tiny op is persisted as it happens. Contrast this with Dropbox, where saving meant occasionally uploading a fresh copy of the entire file; here we never re-send the whole document, we just append the one-character change, which is what makes saving on every keystroke affordable. Presence (cursors, who is online) is the opposite. If we lose it, a cursor disappears for a second and then reappears on the next move. Nobody loses work. Because it is high-churn (cursors move constantly) and worthless once stale, we keep it in Redis in memory with a few-second expiry instead of writing it to a durable database. Writing every cursor wiggle to disk would be a huge waste for data we are happy to throw away. Compaction: replaying millions of ops to open a document would be slow, so periodically we compute the full text at the current version and store it as a snapshot, then we no longer need the ops before it for loading (we may keep them for version history, but loading does not need them). Opening a document then means: load the latest snapshot, replay only the few ops added since it. That keeps open-time fast no matter how old or heavily-edited the document is.
Database Types and Schemas
- Documents (PostgreSQL)
doc_id UUID PRIMARY KEY,
owner_id UUID NOT NULL,
title VARCHAR(255) NOT NULL,
current_version BIGINT NOT NULL DEFAULT 0, -- = number of ops applied; the doc's clock
latest_snapshot_id UUID, -- points at the newest snapshot
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL,
INDEX idx_docs_owner ON (owner_id)
- Operations / Op Log (durable, append-only, partitioned by doc_id)
doc_id UUID NOT NULL,
version BIGINT NOT NULL, -- monotonic per document; the agreed order
author_id UUID NOT NULL,
op_type op_enum NOT NULL, -- 'insert' | 'delete'
position INTEGER NOT NULL, -- where in the text
content TEXT, -- inserted character(s); NULL for delete
length INTEGER, -- chars deleted; NULL for insert
client_op_id VARCHAR(64) NOT NULL,-- idempotency: dedupes client retries
created_at TIMESTAMP NOT NULL,
PRIMARY KEY (doc_id, version), -- ops are read in version order: "WHERE doc_id=? AND version > N"
UNIQUE (doc_id, client_op_id) -- a retried op can never be applied twice
- Snapshots (Snapshot Store: blob storage or DB)
snapshot_id UUID PRIMARY KEY,
doc_id UUID NOT NULL,
version BIGINT NOT NULL, -- the document version this snapshot represents
full_text TEXT NOT NULL, -- entire document text at 'version'
created_at TIMESTAMP NOT NULL,
INDEX idx_snap_doc ON (doc_id, version DESC) -- "latest snapshot for this doc"
- Presence (Redis, ephemeral; NOT a source of truth)
Key: "presence:{doc_id}:{user_id}"
Value: { "cursor": 142, "selection": [142, 150], "color": "#4F86C6", "name": "Ada" }
TTL: ~10 seconds, refreshed on every cursor move
-- if Redis is lost, cursors just blink and re-appear on next move; no document data lost
- Permissions (PostgreSQL)
doc_id UUID NOT NULL,
user_id UUID NOT NULL,
role role_enum NOT NULL, -- 'viewer' | 'editor'
PRIMARY KEY (doc_id, user_id){
"doc_id": "doc_7a2b8f4e9c1d",
"title": "Q3 Planning",
"version": 84213,
"snapshot": {
"version": 84200,
"full_text": "Roadmap for the third quarter..."
},
"ops_since_snapshot": [
{ "version": 84201, "op_type": "insert", "position": 31, "content": " (draft)", "author_id": "user_aa" },
{ "version": 84202, "op_type": "delete", "position": 12, "length": 1, "author_id": "user_bb" }
]
}
// The client renders snapshot.full_text, replays ops_since_snapshot to reach
// version 84213, then opens the WebSocket and starts receiving live edits.{
"type": "submit_op",
"doc_id": "doc_7a2b8f4e9c1d",
"base_version": 84213, // the version the client had when it made this edit
"client_op_id": "c19f-000457",// idempotency key for safe retries
"op": { "op_type": "insert", "position": 18, "content": "h" }
}{
"type": "ack",
"client_op_id": "c19f-000457",
"version": 84214 // the order this op was assigned; now durably saved
}
// The op was transformed against anything that arrived after base_version 84213,
// appended to the durable op log, THEN acked. The ack means "saved", no Save button.{
"type": "remote_op",
"version": 84215,
"author_id": "user_bb",
"op": { "op_type": "insert", "position": 19, "content": "i" }
}
// 'position' has already been transformed by the coordinator so it applies
// cleanly on top of version 84214. The receiving client further transforms it
// against any of its own ops still in flight before applying.{
"type": "presence_update",
"doc_id": "doc_7a2b8f4e9c1d",
"cursor": 142,
"selection": [142, 150]
}{
"type": "presence_update",
"user_id": "user_bb",
"name": "Ada",
"color": "#4F86C6",
"cursor": 142,
"selection": [142, 150]
}
// Ephemeral: never written to the durable op log. If dropped, the cursor simply
// corrects itself on the next move.{
"doc_id": "doc_7a2b8f4e9c1d",
"from_version": 84213,
"to_version": 84217,
"ops": [
{ "version": 84214, "op_type": "insert", "position": 18, "content": "h", "author_id": "user_aa" },
{ "version": 84215, "op_type": "insert", "position": 19, "content": "i", "author_id": "user_bb" }
]
}
// A client that was briefly offline replays these in order to catch up to the
// live version, then submits its own queued offline ops for transformation.Operation (op): the smallest unit of change, almost always a single character. "Insert h at position 18" or "delete 1 character at position 12". We send ops, not whole documents, which is why saving on every keystroke is cheap. Version: a single counter per document that increases by one for every op the coordinator accepts. It is the document's clock and its agreed order. When a client sends an edit it includes the version it was looking at (base_version) so the coordinator knows what the client had and hadn't seen, which is exactly the information needed to transform the op correctly. Transform: adjusting an op so it still does what the user intended after some OTHER op has already changed the document. Concrete example: you mean to delete the character at position 20, but in the meantime a collaborator inserted a character at position 5. Everything after position 5 shifted right by one, so your delete must be transformed from position 20 to position 21 to still hit the character you meant. Doing this consistently on every client is what makes all copies converge to the same text.