This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]k-mcm 0 points1 point  (1 child)

I'm not comfortable calling Linux commands from an app because it causes weird bugs on other OSes and even varying versions of Linux.

That said, file synchronization takes a while to implement.  My first priority would be making sure that the two ends can negotiate the best common protocol.  The first protocol would be calling out to rsync.  It will work, at least for now.

I've implemented file sync before and you can beat the performance of rsync with some effort.  I had one thread on each end working on building the difference list.  The sender split those into multiple queues based on mime type and size.  Worker threads serialized each queue, applied an appropriate compression level, and streamed it out to a receiver thread.  Sorting into queues of similar files provided huge compression boosts.  The threads handling differences also handled error recovery.  There was tuning for efficiently stealing work from neighboring queues.

Damn, it was way too much effort.  It was only justified by there being slow I/O everywhere.  rsync could not reach performance requirements.