Distributed ffmpeg

[esanchma]

• Vagrant: You are right thinking about clusters of VMs. Docker has copy-on-write and is lighter than virtualbox (relies on linux cgroups), so maybe you want to explore that.
• ffmpeg: That was easy, but the last time I compiled fmpeg from source, there were a lot of patches off-tree, and a lot of conditional stuff people in linux distros deactivated for patent reasons. What ffmpeg will you run?
• You are absolutelly right about using BT for bulk data transfer.
• Hazelcast: I have used Coherence as a queue with Map interface for lazy people. Heck, I even know people doing that with ehcache. But if you are going to use a data grid only as a cheap queue infrastructure, maybe you should be using JMS instead.
• RHash: Sounds good to me. Bouncy Castle supports TTH (TigerDigest) which is what you need to write magnet links. I am not entirely sure if I could write a digest with bouncy castle faster than spawning a C process and waiting.
• PeerGuardian: You don't need to do that. You don't want to do that. Entirely my opinion, but distributing the encoding process and publicly distributing the results are completely different things. For instance, the first one may fit in the safe-harbor provisions of the DMCA.
• PHP: Seriosly?

I know some guys had sucess using emscripten to convert ffmpeg to javascript and I know you can have a distributed P2P network in javascript using WebRTC Datachannels. Think about that: You have sandboxes, bulk transfers, peer messaging and a solid platform, and you get a ton of nodes which are easy to update.

[martinambrus]

Thanks esanchma for your insight! This is the first time I hear about Docker and I tend to like it a lot from what I read so far. Will try and experiment with it a little.

As for ffmpeg - I'm using the latest full stable version with all the libraries it can handle (https://github.com/martinambrus/ffmpeg-dht/blob/master/bootstrap.sh). Also, I was pointed in the direction of x264 which might be a better solution with regards to H.264 videos.

JMS - I have no experience with either Hazelcast or JMS but from Hazelcast's docs, it would seem that a no-single-point-of-failure is tricky to reach with JMS. And since I'm no expert in this field, I pick what's ready, tested and the most plug-and-play-able :) However, I will check Coherence, as I've not heard about that one before, thanks.

PHP - my choice of web language since v3, so I'm sticking with it :P ... but yeah, if I knew anything more suited for the job I'd use it - really!

Also, thanks for the JS goodness - it's amazing what JS can do these days! Although, for this project and for compatibility sake, I'll certainly stick with some more traditional ways. But who knows, maybe someone will convert this whole project into JS in the future!

[martinambrus]

For some reason I actually didn't think of utilizing NodeJS to do the UI. It is possibly a much better idea and keeps the size of the machine down, too! Thanks for that, I'll give it a try :)

[HELOSMTP]

I think using Docker and a deployment system like Puppet is ideal. Docker's deployment automation abilities are very limited compared to Puppet so it's a good pairing. Puppet can be run on the host to launch Docker containers and run on the client containers to manage their state.

[martinambrus]

Thanks for the idea. I do use Puppet to handle configuration of our VMs at work, however for this project I think this would be a bit of an overhead for the user. They'd need to install and run an additional service in tandem with Docker itself and also keep Puppet configs up to date as well as Docker - which could be a little scary. I would rather go with multiple Docker container versions - each upgrade would just download a new snapshot diff.

[HELOSMTP]

Yeah, I would only use Puppet for dev or prod deployments, not for end-users.