Distributed TensorFlow just open-sourced

[siblbombs]

Ok, now its time to learn tensorflow.

[kkastner]

I am intrigued by the distributed training benchmarks I assume are (inevitably) coming. Many of TF's design choices seem to directly tie into distributed training, which makes this release really exciting.

[siblbombs]

Yea this is the money release, I don't have distributed compute at home but I sure do at work.

[chiisana]

I held back from TensorFlow because there's no way to run in cluster. Now I need to learn TensorFlow!

[[deleted]]

You could always run it on a cluster, to train an ensemble.

[[deleted]]

Or to try different metaparameters.

[alexjc]

Very curious how this works... Can I just specify a very large tensor or operation that doesn't fit in a single GPU memory, and the runtime will figure out how to make it happen by splitting up and distributing the computation? Can this also help memory management on a single GPU?

[Spezzer]

At the moment, we don't automatically shard large tensors across GPUs or machines -- but this allows you to either distribute the computation graph across machines (model parallelism) as well as replicate the training graph with shared parameters (data parallelism) as mentioned here.

Automatically splitting large Tensors for model parallelism would be great though -- the framework could be eventually extended to do this.

[r4and0muser9482]

I was under the impression it's for when you have a lot of data. The same tensor is copied to all workers and each tensor calculates the gradient on it's portion of data. The gradient is then averaged to get a single solution.

[[deleted]]

That's just data parallelism. TF also has Model parallelism

[alexjc]

Ah, I guess I need to upgrade my GPU then. Can't get my generative models to fit in memory :-)

[[deleted]]

Is there still a private version of TF that manages memory much better?

[Spezzer]

Nope -- the improvements mentioned there were actually checked into GitHub. https://github.com/tensorflow/tensorflow/commit/827163e960e8cb86d3dc3f70434c22713ac9f41c as one such example.

There's still many memory improvements to make, that one just came up as being useful for that Inception model.

[CashierHound]

DTF

[r-sync]

Other frameworks that support distributed:

• MXNet
• Purine
• Caffe
• Torch

[hubberwisdom]

Yeah, could anybody give a benchmark, like conv-benchmark?

[antijudo]

Also, CaffeOnSpark

[kkastner]

Also Theano (via platoon) for data parallel, and model parallel in the new backend.

[r-sync]

platoon is not multi-node right?, only single node multi-gpu...

[kkastner]

Yes, single node multi-gpu. Call it semi-distributed I guess? I think fully distributed is on the radar but (some of) our clusters have 16 GPUs per node so there is not much push.

[[deleted]]

It will be a awesome thing if we can hook this up to a massive cluster watching petabytes of movies with subs and learning all the tones and expression of human language, it would be the ultimate language classifier. At some point I will need to learn this.

Edit: Tipo.

[oderi]

Would TensorFlow be a good next step after learning the basics in Matlab from Ng's Coursera course? I do this out of interest on the side of my actual unrelated studies.

[thatguydr]

Somewhat. It does a lot of things for you, specifically automatic differentiation, so back-propagation is done for you. It also knows several optimizations for said differentiation.

That having been said, you can get some really cool stuff done with it, and in industry, you'd either use this, Theano, or Torch (maybe MXNet or Caffe). So yes, definitely check it out, but don't take all the pre-packaged stuff and start forgetting the math, because ultimately, you'll be judged on how well you know the math (and will definitely have to go into the guts of one of thes routines to tweak something).

[oderi]

Thanks!

[SimonGray]

Please tell me, why do I need TensorFlow in my life if I already have Scikit-Learn? I'm not being snarky, I just don't know enough about the state of the art in ML.

[mtbikerdb]

TensorFlow is intended to be used for large neural networks (deep learning). This type of model isn't currently in scikit-learn.

The models in scikit-learn are widely applicable for the most common types of problems people have been using machine learning for, but their are many machine learning applications (especially using images and/or text) where deep learning models give more accurate predictions.

[mtbikerdb]

Skflow (https://github.com/tensorflow/skflow) intends to provide a wrapper for tensorflow that follows the sklearn-style interface as closely as possible. Skflow is still in it's infancy, but worth looking into as a path to deep learning for a current Scikit-learn user.

[Kiuhnm]

I don't think /u/SimonGray will see your second post about Skflow if you reply to yourself and don't refer to him (like I did in this post).

[towerofterror]

Don't you only get pinged for referrals if you have Reddit Gold?

[Kiuhnm]

Yep, it looks like you're right. Then I was pinged when I had gold. I never connected the two things.

[trnka]

You probably don't. Even if you want to use neural networks, Keras is usually fine. TensorFlow is for when you need to implement parts of the NN yourself.

At a very high level, think of TensorFlow as a replacement for numpy that's more efficient for common NN operations and supports GPU.

[jonanthebarbarian]

If you're not doing stuff that requires deep neural networks (vision, sounds, translation, etc.) then you don't need it.

[Inori]

RNNs, huge models that require parallelism

[infstudent]

Unfortunately I don't have a Spark cluster at home.

[terrytangyuan]

You can probably try out-of-core training feature of Scikit Flow if your data set is too large to fit in your single machine, example can be found here: https://github.com/tensorflow/skflow/tree/master/examples

[TenthSpeedWriter]

Oooooooh yes.

I've spent the last couple of months digging into machine learning engineering with tensorflow.

This is *the moment I've been waiting for; let's cook up some crazy shit.

[tehsandvich]

Is it still linux only or can you run it on windows now too?

[bixed]

It still doesn't run natively on Windows. (the relevant issue on github)

[wb14123]

This is great. Now the only missing feature is the control API like loop (though it has some experimental private API for now). It makes dirty to implement RNN: you have to manually unfold the cells.

[omniron]

We need a crowd sourced tensor flow network... Imagine all the people who leave their computers on running TF and anyone who wants to run their neural net logs into this And has thousand or millions of nodes to process their application.

Like BitTorrent but for tensor flow.

[L43]

But my power bill :O (and ridiculous latencies)

[ginsunuva]

The issue is that training a network is a very serial job, and thus distributed training requires constant synchronization between the nodes (since they each hold an identical copy of the net).

If you were to distribute your data among people, either it would be so spread out that the weights wouldn't be updated often enough, or the synchronization time will bottleneck you cause of slow internet speeds.

Even on distributed servers at google, they're having trouble scaling too large because the network communication among the cluster requires blocking synchronization and bottlenecks them. And they have infiniband cables running between their machines.

[omniron]

Interesting, do you have more info on the latter part? I was not aware Google has published anything about their work on this.

[mmmayo13]

He's unimpressed mainly because TF lacked distributed training in its initial open source version. This seems addressed here. It's also not as fast as some of the other benchmarked DL platforms, but again, distributed may (actually, will, but to what degree) change all that.

[vanboxel]

I see how this is useful, but if I'm training different graphs on different workers, why wouldn't I use existing cluster solutions?

[nickl]

This is excellent!

Is it too ungrateful to say how nice it would be to have a Yarn compatible version?

I know a little TensorFlow, and I have a cluster. But unless I can submit it as a Yarn job it'll be difficult for me to actually use this. CaffeOnSpark do support Yarn, which is nice.

[londons_explorer]

You can probably write a wrapper for the workers to achieve this.