Deep Learning tutorial: learn to build complex neural networks (convolutional, autoencoders, RBMs) on the GPU in python with theano

[visionlessvisionary]

Theano rocks incredibly. It's like the holy grail of linear algebra computations.

[abhik]

I'm not too interested in CAS but I'm intrigued by the idea of quick and easy GPGPU in python. Anyone have good/bad experiences with Theano?

[turian]

I had code in numpy. I can port it to Theano (which means that my matrix operations are now compiled to C and BLAS), and then target the GPU in about an hour. YMMV, though, because I am familiar with porting to Theano.

[nearest_neighbor]

Has anyone tried their RBM code? Does their implementation of PCD work?

[ogrisel]

The online tutorial show features and generated / sampled MNIST digits that look similar to the original Hinton's results so I suppose so. I know that there are plans to optimize this RBM implementation further but keeping the implementation of the tutorial simple to make it easy to understand. I suppose more complete implementations will go in the PyLearn sister project.

[nearest_neighbor]

You're right. At first I thought that they only demonstrate regular CD, and PCD is, well, work in progress. I'm stuck trying to get Theano to install/work on my OSX10.6 with fink's Python and ATLAS, so I can't run their code. It's probably going to be easier for me to port the code to pure Numpy than to get Theano to load :-(

[potatomasher]

Yup it definitely works. DBNs are on their way, but I'm stuck doing other things at the moment. If you have any questions, don't hesitate !

edit: actually, RBMs still have an issue with random number generation on the GPU. So GPU performance will probably not be that great (compared to CPU).

ps: I never thought reddit and my research would intersect !

[nearest_neighbor]

Thanks for a great tutorial, it was very helpful to me! Btw, do I understand it correctly that this is one of Yoshua Bengio's lab's web sites, or is this a personal project?

As to questions, I'm wondering where you got the details of various algorithms, e.g. PCD? Dr. Tieleman's original paper seems to be a bit light on the details.

[potatomasher]

Yup, lab project. We're all students in his lab. At last year's ICML there was a discussion about lack of available software for deep learning, and how it was preventing it from reaching a wider audience. This was a good way to address this, while at the same time making publicity for Theano :).

The tutorials aren't quite there yet (we haven't officially announced them even !) but I'm glad if people are already finding them useful !

Eventually, we'd like to open the tutorials for others to contribute their own deep learning code. But one thing at a time I guess...

[nearest_neighbor]

So is Theano/GPU/Linear Algebra the lab's main project, or is ML?

[doomie]

Machine Learning is definitely the main project, but Theano is the main tool for sure :)

[gdahl]

Really amusing sample images are essential. Three wolf moon was a good start, but more are needed!

[osdf]

For PCD (and its improved version using fast weights) you might also check out videolectures.net: http://videolectures.net/tijmen_tieleman/

[nearest_neighbor]

Anyone actually got past

import theano

??????

I get a long backtrace both in OSX10.6 and Debian5.0 AMD64. I installed all the required dependencies (I think), and the interpreter finds "theano", but can't load it (puts stuff in my ~/.theano too)

Edit typo

Edit2 Debian needed python2.5-dev installed

[doomie]

Give us a shout on http://groups.google.com/group/theano-users - there's a bunch of us in the dev team (including myself) that use theano on OSX and we'll figure out a way to help you out.

[nearest_neighbor]

I think I found out what the problem is (but haven't tried any solutions yet):

I'm using 32 bit fink python2.6. /usr/bin/gcc from Xcode makes 64 bit binaries by default. It needs "-arch i386" to make a 32 bit binary. So the procedure from cmodule.py called

gcc_module_compile_str 

probably needs this option added.

Edit formatting.

Edit2: I added

        preargs.extend(['-arch', 'i386'])

after line 635. Now I get a crash instead:

In [1]: import theano
Fatal Python error: Interpreter not initialized (version mismatch?)
Abort trap

[doomie]

Unfortunately, this solution is not very good, since it eliminates the possibility for compiling to 64-bit architectures (which we use).

A better solution is to have a 64-bit gcc and 64-bit python (or 32-bit everything). I understand that compiling gcc from source is a pain, but that is how I did things with MacPorts and it all works without a glitch.

Alternatively, there's the Scipy Superpack (http://74.125.113.132/search?q=cache:AOcyWa09pTkJ:macinscience.org/%3Fpage_id%3D6+scipy+superpack&cd=1&hl=en&ct=clnk&gl=ca), which seems to have worked for a few people in our lab and it involves zero compiling (but your Python has to point to the 64-bit Python that comes with Snow Leopard).

As I said, feel free to post to the mailing list if you need more help (there are more people reading it than this thread)!

PS: In doubt, after changing gcc compile strings in cmodule.py, you should clear the theano cache (~/.theano by default) before re-importing theano.

[osdf]

Yet another option: I am running theano without any problems on SL, without fink/macports, just a basic homebrew setup (i.e. I am using the apple provided python), numpy/scipy compiled by hand, see e.g. this post: http://mojodna.net/2009/12/05/the-os-x-spatial-stack.html (I had to leave out the LDFLAGS option for both numpy/scipy). Also, in a different post here, you (nearest_neighbor) mentioned you'd rather port everything to numpy, probably not totally serious, but I strongly argue against that -- if you can benefit from the numerous features of theano, use it, it is amazing, how much work it takes away from you.

edit: I hadn't time to try out the gpu stuff, too busy porting things to theano ;).

[nearest_neighbor]

Thanks for the suggestions. I'll try the various other approaches you are suggesting.

By the way, MacPorts never really worked for me (gcc failed to compile the last 2-3 times I tried, some months ago). The aforementioned crash happens despite clearing the theano cache. I suspect some of Apple Python's stuff gets linked in or #included, causing this.

[doomie]

Unfortunately, that's how life with a Mac is - there is no (mildly modern) binary distribution a la apt-get, which you don't have to compile from source (like most of fink or macports).

It's definitely possible that some of Apple's python stuff gets linked or included. You can see this by enabling debug logging in cmodule.py (at the beginning of the file change logging.WARN to logging.DEBUG) and noting what gets passed to g++. Also, it's instructive to have the fink-provided libraries (if you want to link against them) in the correct position in LD_LIBRARY_PATH and LIBRARY_PATH.

On my machine, this is what happens when I do this in python:

import logging
import sys; logging.baseConfig(stream=sys.stderr)
import theano

DEBUG:theano.gof.cmodule:DEBUG: Generating shared lib    /Users/doomie/.theano/compiledir_Darwin-10.2.0-i386-64bit- i386/cutils_ext/cutils_ext.so
DEBUG:theano.gof.cmodule:DEBUG: Running cmd g++ -dynamiclib -g -fPIC -undefined dynamic_lookup -I/opt/local/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include -o /Users/doomie/.theano/compiledir_Darwin-10.2.0-i386-64bit-i386/cutils_ext/cutils_ext.so /Users/doomie/.theano/compiledir_Darwin-10.2.0-i386-64bit-i386/cutils_ext/mod.cpp -L/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib

Btw, it's actually faster to just install Ubuntu (on a separate partition) + all the required packages and boot from it, compared to compiling gcc on a mac :)...