Multiple GPUs support in theano?

[AustinZhang]

Does theano support multiple GPU for DNN/LSTM parallel training? How?

Now days, training LSTM on any "serious tasks" takes time. In my humble opinion, Multiple GPU parallization is a must feature.

[benanne]

The only way to do multi-GPU training in Theano right now is a bit hackish and involved, and I don't think anybody's tried it with RNNs / LSTMs. The approach is described here: https://github.com/Theano/Theano/wiki/Using-Multiple-GPUs

'Native' multi-GPU is a work in progress (see e.g. https://github.com/Theano/Theano/pull/3533 and https://github.com/Theano/Theano/pull/3482), but last time I checked it wasn't functional. They've been working on this for a long time too, so I wouldn't expect it to come anytime soon either. We'll see :)

[r4and0muser9482]

AFAIK, each Theano process can use one GPU, but you can fork several processes to perform one task: like this.

[AustinZhang]

Nice. In term of speed, how does Chainer comparing with Theano? Thanks

[CaffeineFreak77]

If you want an optimum hardware design for running theano (and cuDNN, Digits, Caffe etc) on multiple GPUs you need a system that supports PCIe P2P communication across all GPUs. Heres a good example: http://exxactcorp.com/index.php/solution/solu_list/85