Core Architecture & Concurrency Model:
- Goal: Support seamless multi-user concurrent editing with low latency and eventual consistency without merge conflicts.
1. Concurrency Conflict Resolution:
- Operational Transformation (OT) vs CRDTs: Adopt Conflict-free Replicated Data Types (CRDTs) such as Yjs or Automerge. CRDTs enable decentralized peer-to-peer state merging without requiring a central coordinator to reorder operations.
2. Real-Time Network Transport:
- Establish persistent full-duplex WebSockets between clients and a distributed WebSocket Gateway layer.
- Session Affinity: Route all collaborators on Document X to a shared session cluster using consistent hashing on document_id.
3. State & Persistence Strategy:
- In-Memory State: Maintain active document state in memory on the session host for instantaneous broadcast.
- Change Feed & Write-Ahead Log: Publish operation diffs to an append-only Kafka topic.
- Snapshot Persistence: Periodically compact CRDT operations into periodic document snapshots stored in PostgreSQL or S3 every 30 seconds or 100 operations.
4. Presence & Cursor Tracking:
- Ephemeral cursor positions and selection highlights bypass persistent storage and broadcast over lightweight pub/sub channels.