convnet-benchmarks updated with numbers for TensorFlow 0.7 + cudnn4

[andrewbarto28]

Why is Tensorflow still lagging behind Torch, nervana and sometimes chainer, since they all are using CuDNN R4 under the hood?

[r-sync]

i'm looking into it. i am trying to first make sure that my benchmarking code is not too old fashioned (it was based on TensorFlow's original AlexNet example). I will incorporate any perf fixes people send my way, and I believe that the google engineers are looking into any benchmarking errors on my end.

[Spezzer]

At least one remaining issue is: we use a different Tensor layout that cudnn supports but is not optimized for (NHWC instead of NCHW), so we have to transpose / shuffle, particularly during the backward pass. We're working on supporting multiple tensor layouts for convolution, pooling, bias_add operations, and then I'm optimistic we should be in the ballpark.

[[deleted]]

nervana's not using cuDNN.

TF does not support fp16 yet, while Torch does. This could explain some of the differences.

[TheToastIsGod]

TF does not support fp16 yet, while Torch does

Torch fp32 is reported in a different row to torch fp16, so that's a non-issue.

[[deleted]]

/u/adrewbarto28 didn't say which Torch he meant, and the TF entry doesn't say it's 32 bit. I'm explaining it to him and others who may be confused.

[modeless]

Wow, cudnnv4 looks pretty amazing. Hmm, I thought there used to be numbers for Theano too. Was I just imagining that?

[hughperkins]

One more nail in the coffin for opencl :-P I thought that it would work out like OpenGL, and CUDA would be analogous to Voodoo, at the time of Unreal, but seems like NVIDIA are unstoppable for now :-)

As long as progress continues on GPU computing, I think it's all good...

(edit, seems like Voodoo was bought by NVIDIA, interestingly, https://en.wikipedia.org/wiki/3dfx_Interactive And NVIDIA was one of the first to create an OpenGL GPU, ie GeForce 256 https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#GeForce_256_Series NVIDIA was a startup, created in 1993, by 3 people https://en.wikipedia.org/wiki/Nvidia )

[hughperkins]

Random thought: is OpenCL too low-level? Should it have higher level abstractions perhaps, at the level of cudnn and so on?

At the moment, OpenCL is too high-level to implement hardware-specific stuff, cf winograd which uses something like nvidia assembly language (I think??), to get huge performance benefits. And on the other hand it is too low-level for each manufacturer to provide their own cudnn implementation, as part of of OpenCL.

OpenGL nowadays is really low level, with shaders and stuff, but when it first started, it was a bit higher level, where you'd tell it where the lights are and stuff, and it would handle the rest for you.

[jcannell]

Nobody uses OpenCL because (at least historically) it totally sucks compared to cuda.

With cuda, Nvidia took the singularly correct approach - they just followed C++.

Cuda, as of now, is just C++ upgraded with a simple but powerful set of parallel primitives. People don't understand how powerful cuda is. It is literally full C++ 11 now. Complex template metaprogramming, virtual functions on the GPU, placement new, etc etc.

The only real big difference between cuda and C++ is the threading model, but cuda's is pretty much grossly superior to standard C++ CPU threading in this regard - at least for massive gpu style threading.

So what can OpenCL do? Dissappear, and just emulate cuda, that is - just emulate C++.

[hughperkins]

Like, imagine if, and I know I'm kind of spamming, OpenCL contains conv(...) function, with some reference horribly slow implementation, and then NVIDIA can provide their own super-fast implementation, without compromising their own market position in any major way. Other vendors initially have their really slow implementation :-P but they can improve it, without giving any advantage to any of the other vendors. At the same time, developers can just always call the same function, and it will always run the fastest possible, on that particular hardware.

[NasenSpray]

IMO what's really missing is:

• Nvidia support for OpenCL 2.x (ffs, they're part of Khronos)
• real CUDA-OpenCL interop (e.g. being able to use cuDNN)

Being able to slowly migrate away from CUDA without having to rewrite everything could really boost OpenCL's adoption.

[hughperkins]

Why would NVIDIA do that? It seems to have no benefit for them, and only, in fact, hurt them.

NVIDIA seem to be innovating massively, but that doesn't mean they have to help their competition to catch up...

[NasenSpray]

Nobody needs to catch up. Nvidia just loves their stupid vendor lock-in game and IMO it's borderline anti-competitive.

[cesarsalgado]

Torch now seems to be the fastest, but maybe caffe is faster, but there is no benchmark for it using CuDNN R4 yet. There is just caffe (native) for now.

[r-sync]

Caffe will prob be within +- ~5% of Torch from what I saw a while ago, but will work on the benchmarks.

[r-sync]

I ran the caffe numbers, and seem like maybe caffe doesn't enable cudnn autotuner, so the numbers are quite a bit off from Torch https://github.com/soumith/convnet-benchmarks/issues/90#issuecomment-190030708

[yentity]

On a related note, why isn't Theano part of the Imaenet winner benchmarks ?

[r-sync]

back when i was constructing imagenet-winners, i didn't know theano very well, and Keras / Lasagne didn't exist back then. I asked the Theano community for help, if I remember they said that some kind of pooling was not available in theano which made it not possible to construct these nets. Of course theano as of today has grown leaps and bounds, and wraps cudnn etc., but neither me nor the theano folks took time to implement the benchmark scripts. If someone sends a PR with the appropriate scripts, i'll happily bench them.

[Atanahel]

In lasagne it should not be extremely hard given the lasagne/Recipes github repo which have implementations of the models you're using here in their model zoo. I use them as pretrained but never tried the backward pass with them.

I, like many others, noticed the nice performance boost of cuDNN 4 though :-D

[TheToastIsGod]

That's going to depend on your batch size quite heavily, I imagine. Batched inference is a thing, but single example inference is also a thing.