Collaborative editor (Google Docs)
Let dozens of people edit the same document concurrently in real time, offline or online, and converge to the exact same state. The core design combines Operational Transformation (OT) or sequence CRDTs with an append-only operation log, background snapshot compaction, and ephemeral cursor/presence fan-out.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: WebSocket gateway 1, WebSocket gateway 2
- WebSocket gateway 1 (service) · calls: Document session service
- WebSocket gateway 2 (service) · calls: Document session service
- Document session service (service) · calls: Client opId dedup, Active document cache, Operation log store, Snapshot compaction queue, Document snapshots
- Client opId dedup (dedupStore) · calls: none
- Active document cache (cache) · calls: none
- Operation log store (kvStore) · calls: none
- Snapshot compaction queue (queue) · calls: none
- Snapshot compactors (worker) · calls: Operation log store
- Document snapshots (objectStore) · calls: none
Requirements
Functional
- Multiple users edit the same rich-text document simultaneously and see each other's changes and cursors in real time.
- Strong eventual convergence: any two collaborators who have received the same set of operations display the exact same document.
- Offline editing and reconnect without losing or duplicating operations; version history and point-in-time restore.
Non-functional
- Local typing is zero-latency (optimistic local apply); remote collaborators see edits within ~100–200 ms.
- Zero data loss once an operation is acknowledged; idempotent operation replay across reconnects.
- Cap active editors per document (~100 concurrent writers) while supporting thousands of read-only viewers.
Worked estimates
Operation rate
≈ 2.5M ops/s globally (each ~100–200 B)
5M concurrent active editors × 1 batched op every 2 s
Operation log growth
≈ 32 TB/day of raw ops before compaction into checkpoints
2.5M ops/s × 150 B × 86,400 s
Document load bandwidth
fast O(1) open time instead of replaying millions of historical keystrokes
opening a doc fetches 1 compacted snapshot (~50 KB) + < 100 tail ops
Cursor / presence traffic
ephemeral UDP/WebSocket broadcast — never persisted to the database
20 editors/doc × 5 cursor moves/s × 20 recipients
API
WS op { docId, opId, baseRev, ops[] }
Request: client-generated opId; baseRev is last server revision applied locally
Response: ack { opId, serverRev }; transformed op broadcast to peers
GET /v1/documents/{docId}
Request: optional atRev for history
Response: 200 { snapshotRev, content, tailOps[] }
WS presence { docId, cursorRange, selection }
Request: ephemeral, best-effort
Response: broadcast to active session peers without persistence
Data model
- oplog — store: opstore
- snapshots — store: snapstore
oplog: partitioned by `docId`, ordered by monotonically increasing `rev`. Each entry stores the client `opId`, author, timestamp, and operation delta.
snapshots: periodic compacted document state at `rev = k × 100` in object storage so loading a document only replays the tail (`rev > snapshotRev`).
opdedup: tracks `(docId, opId)` so a client reconnecting and resending unacknowledged edits never applies an insert or delete twice.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Operational Transformation (OT) vs Sequence CRDTs
In Google Jupiter-style OT, a single authoritative server per active document assigns a total order of revisions (`rev = 1, 2, 3...`). Each client holds at most one unacknowledged in-flight operation batch; when an operation referencing `baseRev = 5` arrives at `rev = 7`, the server transforms its character indices against ops 6 and 7 before appending it as `rev = 8`. Sequence CRDTs (RGA, Yjs, Automerge) instead give every character a globally unique immutable ID `(clientId, clock)` and insert relative to left/right neighbour IDs, so operations commute without a central server — ideal for peer-to-peer or long offline edits, at the cost of tombstone metadata.
Single-writer document session routing
For server-ordered OT, routing all editors of `docId` to the same stateful collaboration server (via consistent hashing with leases in etcd/ZooKeeper) lets that server sequence revisions in memory and append to the durable op log without cross-node coordination on every keystroke. When a document exceeds ~100 active writers, additional joiners are placed in a read-only follower fan-out tree fed by the session server.
Operation log compaction and version history
Replaying 500,000 operations from `rev = 0` on every page load is too slow. Every N operations (or when a session becomes idle), a background worker folds the tail operations into a new snapshot at `snapshotRev`. Named versions and history scrubbers read the nearest preceding snapshot and replay forward to the target revision.
Failure scenarios
WebSocket acknowledgements are dropped during reconnect
Scenario and expected result
From t = 1.5 s to 1.58 s responses from the collaboration service to gateway 1 are lost; the gateway retries the buffered operations with the same `opId`.
Expected: The dedup store catches the retried `opId`s and returns the original revision ack without appending duplicate text to the document log. Compare the no-dedup alternative.
Snapshot compaction workers crash
Scenario and expected result
The background snapshot compactors are down from t = 1.3 s to 2.0 s.
Expected: Live editing and document loading keep working at 100% goodput: edits only need the operation log, and loads read the last snapshot plus a slightly longer tail of operations until compactors catch up.
Everyone opens and edits the company all-hands doc
Scenario and expected result
One hot document receives 30% of all load and edit traffic.
Expected: The active document cache serves document loads, and routing the document's session to one collaboration server serialises its revision log cleanly in memory.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No operation deduplication on reconnect
- When a WebSocket drops after the server appends an operation to the log but before the client receives the `ack`, the client resends its unacknowledged buffer. Without `opId` deduplication (or idempotent CRDT item IDs), the character insertion is applied twice.
Reference
- Inspect the architecture above and run the same scenario.
Write full document snapshot to object storage on every edit
- Rewriting a 100 KB document blob on every keystroke instead of appending a 150 B operation to a log amplifies write bandwidth by ~700× and pushes p99 latency above 80 ms.
Reference
- Inspect the architecture above and run the same scenario.
Pessimistic locking (check-out / check-in)
- Locking the document or paragraph while one user types avoids OT/CRDT complexity entirely, but blocks real-time collaboration and leaves stale locks whenever a browser tab disconnects.
Clarifying questions
- How many concurrent editors vs read-only viewers must a single document support?
- Do we need long-lived offline editing with automatic merge, or primarily online collaboration?
- Is the document plain text, rich text (spans), or block-structured (Notion/Figma)?
- What granularity of version history must be retained?
Common mistakes
- Sending the entire document text on every keystroke instead of compact operations.
- Applying `insert(pos, char)` from concurrent editors without OT index transformation or CRDT IDs.
- Persisting every cursor movement to the database.
- Replaying the entire operation log from revision 0 on document open without periodic snapshots.
- Omitting client operation IDs, causing duplicate insertions when a client reconnects.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Edit a shared document.
- 2/4 Real-time sync + cursors.
- 3/4 Convergence guarantee, optimistic local edits, offline reconnect, and writer caps.
- 4/4 Distinguishes persistent op log from ephemeral presence and read-only viewer fan-out.
estimates / levels 1-4
- 1/4 None.
- 2/4 Active users.
- 3/4 Op rate, op log growth, and snapshot compaction intervals.
- 4/4 Separates persistent edit bandwidth from quadratic cursor broadcast traffic.
api / levels 1-4
- 1/4 Save document.
- 2/4 WebSocket ops.
- 3/4 Client `opId`, `baseRev`, server `ack` with `serverRev`, and snapshot+tail load.
- 4/4 Single in-flight batch protocol (Jupiter) and reconnect catch-up.
data model / levels 1-4
- 1/4 Documents table.
- 2/4 Document blob + history.
- 3/4 Partitioned append-only op log + periodic snapshots in object storage.
- 4/4 Tombstone/compaction metadata and client opId deduplication.
high level / levels 1-4
- 1/4 Clients write DB.
- 2/4 WebSocket server + DB.
- 3/4 Session routing per docId, op log, snapshot compactor, presence channel.
- 4/4 Read-only fan-out tier for viral docs and lease-based session failover.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Names OT or CRDT.
- 3/4 Explains index shift in OT vs unique character IDs in RGA/Yjs CRDTs.
- 4/4 Jupiter single-in-flight state machine, tombstone GC, and rich-text formatting spans.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions latency.
- 3/4 Session server crash and divergence bugs as top risks.
- 4/4 Automated fuzz/convergence testing, op-log lag alerts, and session failover drills.