DavidCloud
Overview
DavidCloud is a fault-tolerant, distributed cloud platform that provides webmail, file storage, and user management on top of a replicated key-value store. The system is split into four subsystems: a frontend load balancer, a pool of frontend web servers, a backend coordinator, and a set of KVS storage workers. A client first connects to the load balancer, which redirects it to a live frontend server. That server parses the HTTP/1.1 request, authenticates the session, and for every storage operation asks the coordinator which worker owns the data before reading or writing it. Workers execute operations against an in-memory store, replicate them to secondaries, and durably log everything for crash recovery. I worked on the cloud storage layer (DavidDrive), the key-value store, and the load balancer.
Storage Engine
At the bottom of the stack is a distributed key-value store. Each worker holds an in-memory nested hash table that maps a row key to a set of columns, and the key space is partitioned into tablets, the unit of ownership and replication. The coordinator decides which worker owns each tablet using consistent hashing with virtual nodes: every physical node is hashed into 64 positions on a 64-bit ring, and each tablet walks clockwise from its own hashed position to pick a primary and its secondaries. The virtual nodes keep the distribution even and, more importantly, mean that when a node joins or dies only a proportional fraction of tablets move rather than the whole key space being reshuffled.
Frontends never talk to a worker blindly. Before each operation they send the coordinator a
LOOKUP for the row key, which hashes it to a tablet and returns the current
primary plus the ordered list of live replicas and the cluster epoch. This keeps writes
always flowing to the correct primary even as ownership shifts underneath them. Concurrency
on a worker is handled with cell-level locking (a reference-counted mutex keyed by
row + col), so contention stays fine-grained, and row deletions lock affected
cells in sorted order to avoid deadlock.
Replication and Fault Tolerance
Every mutating operation runs through a two-phase commit between the primary and its
secondaries. The primary first confirms it still owns the row, acquires the cell lock, and
writes a PREPARE record to its write-ahead log. It then sends PREPARE
to all secondaries in parallel; each one validates the operation locally, logs a
PREPARED record, and replies ACCEPT. Only once every secondary
accepts does the primary log COMMIT, broadcast the commit, apply the update, and
acknowledge the client. Every WAL record is fsync'd before any network or memory
state changes, so the system can always recover a consistent view after a crash.
Liveness is tracked through heartbeats: each worker pings the coordinator over a persistent connection, and a reaper thread marks any node silent for more than four seconds as dead. Any liveness change bumps the epoch and triggers a tablet reassignment, so a failed primary is seamlessly replaced by a surviving replica; the cluster keeps serving as long as one node in a replica set is alive. The coordinator also drives a global two-phase checkpoint every 20 seconds: workers gate new writes, serialize their tablets to disk, rotate their WALs to reclaim space, and then reopen the gate.
The trickiest part was recovery for a rejoining node. The naive approach, scanning local checkpoint and log files to figure out what the node used to own, is wrong, because the hash ring may have shifted while the node was down, leaving it responsible for tablets it has no local state for. Instead, before re-registering, a recovering node queries the coordinator for the current tablet assignments, live endpoints, and replica count, then rebuilds the consistent hash ring locally with itself included to compute its intended tablet set. For each tablet it compares its checkpoint against the primary's: matching checkpoints only need the missing tail of the WAL replayed, while a stale checkpoint pulls a full snapshot. Only after every tablet is synced does the node start heartbeating and accepting traffic.
DavidDrive File System
On top of the key-value store sits DavidDrive, a per-user hierarchical file system. Directories and files are modeled as inodes layered over the KVS (each directory row holds its child entries and each file row holds metadata), giving users folder navigation, rename, move, and delete with normalized path resolution that keeps every user isolated inside their own root namespace. Small files are stored inline, while files up to 500 MB are transparently split into fixed-size chunks spread across the storage cluster; metadata records the total size, chunk count, and content type so downloads and inline previews can stream the chunks back and reassemble the original file. Because every file operation goes through the same coordinator lookup and replicated write path, the file system inherits the store's consistency, logging, and recovery guarantees for free.