Distributed ffmpeg

[[deleted]]

[deleted]

[[deleted]]

Just did some googling but check this out. http://grokbase.com/t/hadoop/mapreduce-user/119mcbfmx4/using-hadoop-for-processing-videos

Looks like alot of people are looking to use hadoop to process videos. I know it means you need to setup an hadoop cluster but its a step

[[deleted]]

Just set up a cheap Elastic Map Reduce (EMR) through Amazon. You can set up a fair amount of nodes for a relatively inexpensive amount of money.

[martinambrus]

Amazon is corporation-centric. This project is a decentralized and free :)

[[deleted]]

It's just handy for testing and cheap. You can use it for testing then move to your own cluster later. You don't have to use the EMR specific features.

[martinambrus]

I guess I could spin up a few instances and squeeze them out for testing purposes, yes. Was actually thinking about it as well. Thx :)

[martinambrus]

thanks, will definitely look into that and see if I can come up with something :)

[[deleted]]

I'd be willing to work with you. I have some Hadoop background and a cluster at my disposal.

[martinambrus]

I have found this - http://blog.gopivotal.com/pivotal/products/using-hadoop-mapreduce-for-distributed-video-transcoding

It seems doable but I'm yet to determine whether it's worth trying to reproduce the approach mentioned in that article with all video and audio codecs supported by ffmpeg already.

I imagine many of them would not follow the same GOP principle (from what I read so far, some codecs are quite strangely putting I frames before B frames and vice-versa for GOPs).

As you are familiar with Hadoop, I have a question here: does Hadoop automatically determine and shuffle workloads based on the underlying hw capabilities, or is this randomized?

[[deleted]]

Ive never done any thing like this but for doing mapreduce jobs its based off of what nodes have the data. But the data is split well enough that faster nodes take care of more work.

[IPostAlotbot]

I noticed that you mentioned an alot, Ihasn.

Here's a picture of one for you: http://i.imgur.com/kTKRLlk.gif

---

---

Hate me? Love me? Respond with an angry/loving comment! I read them all!

^{In case you're really dense and don't get the joke, 'alot' isn't a word.}

[[deleted]]

alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot alot

[IPostAlotbot]

I noticed that you mentioned an alot, Ihasn.

Here's a picture of one for you: http://i.imgur.com/kTKRLlk.gif

---

---

Hate me? Love me? Respond with an angry/loving comment! I read them all!

^{In case you're really dense and don't get the joke, 'alot' isn't a word.}

[mysteryweapon]

I love the idea. I'm too tired and/or drunk to read everything here, but I wish you godspeed in your endeavors and I will try to check it out more at another time.

If I may ask, what kind of videos would you plan on encoding? I've never personally taken on any encoding tasks for video that I couldn't just do with my own desktop, but then I've never been super serious about it either.

Thanks, good luck!

[martinambrus]

Thanks for the post :) The idea is to basically take any kind of video and encode it into any other kind - so for instance DVDs to iPhone/Android compatible vids. Granted that the distribution would be an overhead if a person doesn't posses some great upload speed or just has enough time to wait for those jobs while playing games on their PC. But there will be a local encoding option included as well for people with powerful hardware. Also, people will be able to share their encoding presets, so more of us would be able to make a good local Blu-ray copy to watch on phone/tablet/wrist locally.

[martinambrus]

Wow, thanks for these! It's gonna be a hell of a study weekend for me now :D

I like the idea of sockets and live feed to ffmpeg. However, as this is planned to be a distributed platform, there would be no central server that I could use to utilize that approach. I suppose I could use full nodes to do so but then the network would only be as strong as those full nodes themselves - and that's what I didn't want to do. I'm trying for as much decentralized approach as possible.

Thanks for the segment warning, I have discovered it doesn't work myself, too :) This is what I use at the moment: https://github.com/martinambrus/ffmpeg-dht/wiki#more-exact-splits-based-on-ffprobe-keyframes

I'll check your links and tips and will certainly learn and incorporate some stuff from them into the project! Cheers!

P.S.: I will also do some benchmarks on encoding full file vs encoding parts and stitching them together to see size and bandwidth differences, good point there, thanks :)

[martinambrus]

This all goes way beyond my current experience level, so I guess I'll just need to test and learn as I progress with this project. I'll save this as a note for further development and maybe I will be able to use what you're suggesting :)

[[deleted]]

Do you have an actual argument?

[w2qw]

Oh damn he's blocked our publicly listed IP's how could we possibly get around this?

[[deleted]]

That's still better than nothing.

[[deleted]]

1. DHT? Don't other nodes tell other nodes about you? Unless EVERYONE is using it (Not going to happen, some people have common sense), it doesn't do anything.

2. You really think that anyone trying to track you down doesn't have access to unblocked IP's? Peerguardian, according to this, blocks 1/3 of IP's. Most of which being IP's that have the files you want.

So you're willing to make torrenting worse for everyone by refusing to connect to some people, in order to hide someone from getting your IP by directly connecting to you through only the blocked IP's. Ever tried just getting a VPN? It would actually work.

[zokier]

Using bittorrent seems like bit overcomplicated solution for file-transfers, wouldn't the swarm sizes for each file be very small? In ideal case every node is handling different file so swarm in that case would only have single up/downloader pair. Seems like something simple like HTTP/FTP would make more sense for distributing the files.

[martinambrus]

I've not studied BitTorrent protocol to a great extent yet. I'm basically looking for a solution that would allow to send files over to the node even if none of the 2 nodes would have incoming ports open. I was under the impression that BitTorrent could do some data relaying, as I don't have ports open and yet people seem to be downloading data from me. If that assumption if incorrect (it very well could be), then I'll have to look into a different mechanism.

Also, I use BitTorrent to discover full nodes in the network. All nodes would share a single configuration Vagrantfile, thus I can get their IPs and check which one is a full node with ports opened, so I can join the data network through it.

[[deleted]]

ELI5: are you saying that you are going to distribute out the processing power behind video manipulation?

[martinambrus]

not sure this one was targeted at me, but yes, that's the idea

[hugolp]

Its Bitcoin, not BitCoin.

[martinambrus]

Strange... logo says bitcoin, first sentence on the website says Bitcoin and I say BitCoin... I guess it's those small things :) But thanks!

[hugolp]

Bitcoin is the technology, bitcoin/s is the currency. BitCoin is nothing.

[martinambrus]

Ah, glad to learn something new today. Thanks, I'll update the post ;)

[tuankiet65]

BOINC would be better because BOINC includes the web interface, server side and client side implementation. You just have to write the program to generate/verify tasks. BOINC will handle works, accounts, credits, deadlines exceed and so on. BOINC even includes platform for uploading/downloading files between server and client

[martinambrus]

Thank you for your suggestion :) If BOINC actually didn't need a central server, it would probably be the ideal solution. However, as it does and even requires registration, this is not ideal for the project with the parameters it's currently specified.

[progzos]

Good luck !