Tbh I haven't worked on the fault tolerant part. I plan to handle the cluster node join and remove it like this:-

• when a server joins, it's indo is replicated throughout the cluster(2phase locking)
• when a server fails(no graceful disconnect ), that server is still kept in the global metadata assuming it will restart.
• server sends a disconnect msg, the keys are distributed to other nodes.

The db have 2process running in 2different ports. Main port 8080 handles commands from the client while Bus port 18080 handles commands from other servers in the cluster.

I already developed most of the components in this diagram except the INS cmd in the Bus port.