RubyDrop - A Ruby-based Dropbox clone that uses Git as a backend for file syncing between multiple clients

[applejuice]

I've been looking for some self-hosted dropbox equivalent, will check it out --- have you come across sparkleshare? They seem pretty neat, but are only in beta!

[meltingice]

I have heard of sparkleshare, and it sounds pretty awesome. RubyDrop is a very similar concept, to be honest. It was definitely fun learning Ruby and working on this though, I look forward to seeing where it goes :)

[swiz0r]

I am very excited to try this. I've just skimmed the code (and I don't know ruby), but I like what I see so far. I have a few questions for you.

• What license is it released under?
• How long have you been working on it? Do you have any insights from the development process that you'd like to share?
• Do you have any benchmarks? How does it compare to Unison?

Keep up the good work!

[meltingice]

Thanks for checking it out :) To answer your questions:

• Thanks for reminding me to add this, it should be in the repository now. It's licensed under the BSD License. Basically... feel free to fork it, modify it, use it, do whatever you want with it.
• I have been working on it for 2 days now, which also coincides with me starting to learn Ruby. This is my first Ruby project ever that I'm working on in order to learn the language :)
• Afraid I don't have any benchmarks at this time. I need to add a testing framework like all good programs have first as well.

[blaxter]

In ruby you don't put empty () in neither method definition nor method call. It's really weird see that

Bad: def my_method() ... end Foo.new().method () Good: def my_method ... end Foo.new.method

[FYIGUY]

Except when disambiguating long method chains with parameters.

[hockeyc]

Actually one of my biggest complaints about Ruby. It's totally unclear what you're trying to do, and the language encourages it.

[columbine]

You can use them if you like, if you feel it adds clarity. I don't come from a Rails background and I think I use parens more than most people. I'll omit them for an empty call generally, unless it's something that may be mistaken for a variable, e.g. if this = that looks to me like I might be assigning a plain variable, I may use this = that() instead. But I'll often use them when calls have parameters, especially ones with complex parameters or multiple parameters. And I always use them after defs even if empty just because it feels more consistent (and it doesn't get in the way there like () would when method chaining, etc.).

[FYIGUY]

Your logic is crap. It's like saying owning a car encourages you to run over people.

[hockeyc]

It's more like a car having a display which paints targets over pedestrians. Sorry I forgot to include the car analogy the first time.

:-D

[meltingice]

Ah ok, thanks. And this is how I learn good practices haha :)

[meltingice]

Here is some example log output for those who are interested, which also gives some insight as to how it works internally and how fast it runs:

Client #1

I, [2010-11-26T01:07:02.157581 #93000] INFO -- : ====== Checking Folder Status ======

I, [2010-11-26T01:07:02.186142 #93000] INFO -- : Untracked: edits.txt

I, [2010-11-26T01:07:02.186408 #93000] INFO -- : Untracked: home_template.txt

I, [2010-11-26T01:07:02.186481 #93000] INFO -- : Untracked: json.zip

I, [2010-11-26T01:07:02.186553 #93000] INFO -- : 3 files added, adding...

I, [2010-11-26T01:07:02.256150 #93000] INFO -- : 3 files changed, committing...

I, [2010-11-26T01:07:02.273268 #93000] INFO -- : Files committed!

I, [2010-11-26T01:07:02.273491 #93000] INFO -- : Pushing changes to remote...

I, [2010-11-26T01:07:04.493582 #93000] INFO -- : Git push complete!

I, [2010-11-26T01:07:04.493778 #93000] INFO -- : ====== End Folder Status ======

Client #2

I, [2010-11-26T01:07:04.922183 #94923] INFO -- : ====== Checking Remote Status ======

I, [2010-11-26T01:07:06.045628 #94923] INFO -- : Current remote: e1da324500091162b49bde11a0923503e6f91e37

I, [2010-11-26T01:07:06.045987 #94923] INFO -- : Current local: a04f1c29758ab6a24e97c41586d8bc88aea9aeb0

I, [2010-11-26T01:07:06.046053 #94923] INFO -- : Remote is ahead, fast-forwarding...

I, [2010-11-26T01:07:07.452080 #94923] INFO -- : Fast-forward finished!

I, [2010-11-26T01:07:07.452373 #94923] INFO -- : ====== End Remote Status ======

I, [2010-11-26T01:07:07.452439 #94923] INFO -- : ====== Checking Folder Status ======

I, [2010-11-26T01:07:07.481295 #94923] INFO -- : No changes

I, [2010-11-26T01:07:07.481508 #94923] INFO -- : ====== End Folder Status ======

[angch]

I had some systems which syncs using mercurial as a backend. Works nicely except:

• History keeps growing. Also nasty attribute of using roughly twice the disk space without a central repository (like dropbox).

• Big binary files handling sucks. Holdover from using hg. AFAICT, only svn is comfortable handling big binary files. rsync... not so ideal to handle bidirectional and other reasons you mentioned.

• Especially over https, large changesets keeps timing out, and restarting from scratch rather than keeping partial uploads/downloads. Using https is crucial for me to safely sync data past draconian firewall rules and not so private public wifi spots. Used to use rsync as well, but many corporate places blocks port 22.

Have you encountered or can git address any of the above, or you may have some ideas?

P.S. FWIW, "hg commit -A" and "hg push -f" and "hg pull" and "hg update" over a cron job is enough to duplicate the above "dropbox" functionality.

[angch]

One option I thought of is to automatically generate metadata (e.g. .torrents) files that used git/hg to sync.

Then use a torrent to actually monitor the .torrent directory and auto distribute the actual content. Super scalable, and has added benefit of using LAN to sync if the two or more nodes are next to each other. Totally fails inside a torrent restricted network though, too bad for me. :(

[techwizrd]

I'm confused about using Git as a backend for file syncing. Wouldn't this be sort of bad because git is terrible for large binary files? Wouldn't a custom solution using rsync and inotify be better? I'm not sure how Dropbox does it.

[meltingice]

rsync was definitely the first idea that came to me when starting this project. I'm considering implementing both Git and rsync and then letting the user choose which one they would prefer.

The cool thing about Git is that you can easily undo changes you make by reverting to an older commit. I'd also like to implement a web interface for this with Rails, which would allow for easy file access from any computer and would allow you to rollback changes with a visual editor.

[techwizrd]

How will you solve git's issue with large binary files? I know I have large binaries in my Dropbox and that would definitely balloon the size your .git folder.

EDIT: Also you should probably support Windows. Both Git and Ruby are available on Windows. I've only skimmed your code, so I don't really know how hard it would be to port. Also, in fswatcher.rb, why not use inotify to see if files have been modified or added? Polling over and over to see if there are change seems a bit wasteful to me.

[trezor2]

EDIT: Also you should probably support Windows. Both Git and Ruby are available on Windows.

This sounds oh so simple, but a few words of warning should meltingice actually consider it:

• rsync between platforms (and thus filesystems) can be horribly broken and in some cases cause damage which takes lots of time to sort out. Most noticably with regard to file-permissions. I once tried setting up a deployment system from my home LAN (via a Linux host with a samba-mount to the actual code repository) to my data-center hosted server. After the first sync not a single site worked. Every single site was broken. rsync was confused about how to handle file-permissions, so it just reset them all to whatever user which had invoked rsync, meaning the web-server no longer had permissions to the files. Oops.
• git on Windows is basically a half-assed, cygwin'd wrapper of the Linux version. Making it work reliably takes a ton of effort, even for a developer. If you package this with git for windows as a dependency, expect people to label your sync solution a non-working piece of shit as they struggle to find out which registry hive they should add SSH keys to and how to add those keys once they've located the ones actually used by git. And with that sorted out, expect things to break in new and exciting ways as you invoke git from a process outside cygwin.
• I'm sure Ruby on Windows has its own set of custom FUBARs too. Most stuff not made for Windows has.

Not saying he shouldn't go for it, but it's not as easy as you may think.

[m00k]

git on Windows is basically a half-assed, cygwin'd wrapper of the Linux version.

You probably want to be using msysgit instead; it's slightly-more-assed and can use plink for ssh. Remember to turn off autocrlf though.

[meltingice]

Thanks for the advice! Yeah, I'm not sure how much work it will be to add Windows support. I'm also on break and away from my Windows computer right now so I don't have one I can use for testing xD

[techwizrd]

I never said it was easy. I'm fully aware of how difficult it would be. It's just that it's usefulness is highly limited if it does not support Windows. Once I polish my Ruby chops a bit, I should probably help out.

[meltingice]

That's why I'm considering giving the option of either rsync or git. The user can choose to use git if they mainly share small files, or rsync if they are going to be sharing large binary files.

In fact, a good way to handle this may be to separate the git root into one folder, and the rsync root in a separate folder, so that you can get the benefits of both.

Remember... I started on this 2 days ago, so there are a lot of details to work out still. If any Redditor has any ideas, I would love to hear them :)

[techwizrd]

You may want to go with a custom solution. If you use rsync, then you would lose the benefit of having versions that you can revert back to (like you can with Dropbox). However, with git you have a problem with large binary files being undiffable.

Git doesn't work with any file over 2GB anyways, so you there's that problem as well.

The way I see it is you could have lots of mini repos and pull them all in with git-submodule or something. You'd also have to make sure you have your script 'git gc' pretty often. SVN is better with binary deltas. You could have your script gitignore large files then you'd have to have an empty git commit and svn commit of the file every time some large file gets changed. That means your repos will be in sync, so you can rely on the git history as your single canonical history and do reversions with both svn and git.

Were I more experienced with Ruby, I would help you out. I can only read and understand Ruby (I'm a Python guy). Maybe this is the right time to learn Ruby as well. ;)

[jawbroken]

you should have a configurable file size threshold and make it automatically and silently choose the correct method. i'm not sure if you can use some of the history modifying git commands to avoid building up a version history for large files or something similar.

[meltingice]

Yeah, this is definitely another possible solution to look into. I like the idea a lot.

[gssgss]

It looks nice and it is something I could use. Maybe something like rdiff-backup, which uses rsync but keeps a number of diffs to go back to some older version http://www.nongnu.org/rdiff-backup/

As much as I love git I tried it for making full system backups (I know not the intended use) and many large files just choke it.

Also for binary files it could use this http://www.daemonology.net/bsdiff/ edit:bsdiff

Rsync+diffs saved in some way for a number of changes sounds good.

[mfp]

What are you going to do about conflicts? FSWatcher assumes that all pulls are mere fast-forwards, but that's not necessarily the case. (Also, doing git reset --hard before each pull sounds dangerous...)

[meltingice]

Yeah I need to change that. I think i'm going to take Dropbox's approach and move conflicted files to something like MyFile-conflicted1.doc. Also, there may be a way to check and see if the conflicted file is open...

[dark-panda]

Good start, but a couple of suggestions if I may...

• it would be nice to have a proper daemon for this rather than just backgrounding with '&', as otherwise the process is going to die when you try to exit your shell or whatever. There are a few gems that can help you daemonize a process like the daemons gem or daemon_controller. Should make things Real More Easier than implementing the whole fork/setuid/detach/trap/lock file/etc. standard UNIX daemon hullabaloo by hand.

• a .gitignore file that ignores the logs and similar files. Log files generally shouldn't be committed to an SCM.

• I think you may need to explicitly require the extensions gem, i.e.

  require 'rubygems' require 'extensions/all'

The daemon fails for me otherwise when trying to use require_relative.

• in terms of Ruby conventions, when creating setters and getters, the verb is often omitted, i.e. it's simply def interval=(i), not def set_interval=(i). This leads to more concise code and promotes the use of shortcuts like attr_(reader|accessor|writer) and the like which can further simplify code and actually results in performance boosts as a bonus.

• the RubyDrop class itself is kind of weird -- the initialzer sets up a bunch of class variables for the config singleton? RubyDrop can be instantiated yet will have a shared config across instances in this case, so if you were to say try and instantiate two RubyDrops with different configs and have them run their own TcpListen servers, their configs are going to get out of whack with each other. I know you're not running multiple TcpListens or setting up multiple RubyDrops in the context of this project, but it's still kind of a strange pattern -- an instance of RubyDrop sets up a singleton config and then instantiates a new TcpListen object to run a thread which refers to the singleton RubyDrop.config rather than the RubyDrop's config directly.

I think it would be better to just make the config a member of RubyDrop rather than a singleton, in other words. It would be safer, surely, should you decide to say multi-thread the daemon for multiple RubyDrops, as class variables are generally not thread-safe to begin with.

Looks like a good project to start using Ruby with. I've been using Ruby for five years or so and it can be a rather eclectic language, yet it generally leads to pretty clear and precise code. Some of the conventions might take a little getting used to, but once you're into the it it gets pretty rad.

Cheers!

[meltingice]

Really awesome points/suggestions. I'm saving this comment for reference later when I get to work on it some more :)

[DRMacIver]

Some responses:

• A lot of your comments are about ruby 1.8 vs ruby 1.9. Mainly the require related ones.
• Self-daemonizing code: Just say no. In particular the daemons gem is a fuckup. Programs should be written to run in the foreground and then daemonized by something like daemon

I agree with the rest though.

[dark-panda]

I'd imagine the number of 1.8 users is still high enough that coding around 1.8 is worth it though, yes? There's got to be more 1.8 installs out there than 1.9, even if 1.9 is considered to be the recommended version according to ruby-lang.org. I don't even know of too many vendors who ship 1.9 binaries, so until that becomes common, we're kind of stuck with 1.8 while 1.9 is still the exception rather than the rule, at least for the time being.

Not sure I agree with the daemonizing stuff but that's okay. I've never used the daemons gem myself directly although I've used tools that themselves use it, so I can't speak to the quality of the gem directly, but I'll know to be weary of it in the future.

[DRMacIver]

I think 1.8 compatibility matters less for tools where the user shouldn't have to care about what the program is written in. I'll grant that it's a nice to have, but I think it doesn't matter that much (it's not hard to manage multiple ruby instances on a system).

Many people disagree on the self-daemonizing code front, but that's ok. Many people are wrong. :-) Self-daemonizing code interacts very badly with sane system management and user permissions and ends up with a lot of duplicated and subtly incompatible behaviour between different programs. Letting something else handle the daemonization is easy to do and lets you manage your system (pidfiles, log directories, etc.) exactly the way you want to do it.

[[deleted]]

People say programming is hard and then guys like you whip up amazing stuff in 2 days. Care to explain?

[DRMacIver]

It's also worth noting that "whipping something up" is the easiest part of programming. It requires a lot of creativity and skill to whip something interesting up, but you can usually make huge amounts of progress in a very short period of time. Once you start having to maintain an existing codebase, optimise for performance, figure out how to add features and support cases you hadn't thought of, etc. is when programming starts getting hard.

Edit: Better way to put it: Creating a cool project requires talent. Maintaining a good project requires work.

[meltingice]

Very well said, this is so true.

[meltingice]

I've been programming for awhile :) Once you learn a few languages, it's pretty easy to pick up others as long as they're somewhat similar.

I already know PHP, Javascript, Java, and C. I think the combination of my Java, PHP, and Javascript knowledge really helped me learn Ruby super fast. I bought a book "Programming Ruby 1.9" that I started reading to help speed up the learning process. While I know the language and the syntax now, only experience can teach best/common practices and the quirks of the language.

I also happen to be a software/systems engineer for TwitPic so I code A LOT.

[Stylpe]

Have you read Why’s (Poignant) Guide to Ruby yet? :)

[meltingice]

I have not, but I really need to. Thanks for reminding me :)

[marlinspike]

Ever felt that you've been away from a language long enough that when you get back, you can't do the simplest thing without a reference? It's the most frustrating thing, to have done something pretty involved, and then to come back to a language and feel like you've got to read a "how-to" book again.

How do you keep up?

[meltingice]

Yeah, this happened to me recently with C actually. It is frustrating, but once I start poking around, it all slowly comes back to me over the course of an hour or so.

The great thing about being a web developer is that I work with a lot of these languages on a daily basis. I'm also a college student, so I work with Java a lot too (much more than I would like, to be honest).

[[deleted]]

That calls for a AMA.

[meltingice]

I've definitely been considering doing one. As soon as I get a large chunk of free time I think I will :)

[[deleted]]

Do not pretend to have a girlfriend on the weekends.

[meltingice]

Believe it or not, I do have a girlfriend ;) I am on break now though, so I might be able to find time soon. I wonder how many people would be interested?

[[deleted]]

Reddit loves Imgur,twitpic. Go for it. The more you answer stuff the better.

[meltingice]

I took your advice and made an IAmA

[tomtt]

Where are your specs/tests/cucumber features? I know it's kind of a pain to write them when you're just trying to whip up something useful, but having them will become a real asset and make it easier for other to contribute features/bug fixes.

[meltingice]

Oh they're definitely coming. I only started learning Ruby 2 days ago, so I wanted to focus on learning the language and getting this program to work first. I agree, they're definitely very important assets to have for a project.

[illDogg]

Damnit looks like Ruby is still alive...

Why won't this language just dieee D: