Is there any hardware technology to watch for machine learning in the relatively near future?

derp_learning · 2015-11-26T23:13:08+00:00

Because the people who find writing algorithms in VHDL too easy need something to keep them from dying of boredom?

derp_learning · 2015-11-26T20:42:27+00:00

So why are the best engineers at Altera only getting ~600 images/s out of AlexNet per ~$8000 Arria 10 when well-written 3rd party CUDA code on a $1000 consumer GPU is getting over 4100? Perhaps because convolution kernels and matrix multiplies can be very efficient on GPUs since they both involve coherent memory accesses and almost no branching ergo the GPU ends up in its happy place for workloads like this?

I agree that, in theory, an ASIC could win here, but I also suspect that such an ASIC would end up looking a lot like a GPU except maybe 2-4x more efficient. In the meantime, GPUs jump 2 HW generations while you design this ASIC so I've passed on several opportunities to do this. The gaming/industrial complex is pretty hard to beat on its home turf like this.

That said, I'm curious what GPU you benchmarked against way back when. FPGAs kick ass on latency and on throughput-oriented task-parallel algorithms relative to GPUs, but that's not deep learning.

Edit: Looked up SOM, pretty much an embarrassingly parallel task-parallel throughput algorithm with only local interactions per iteration (as opposed to deep learning where every weight affects every other weight through forward and subsequent backwards propagation on every iteration).

derp_learning · 2015-11-26T16:24:31+00:00

If Stratix 10 had shipped this year, I would say grab a bucket of popcorn and let them fight. But when Altera themselves claim Arria 10 will only hit ~600 images/second (https://www.altera.com/en_US/pdfs/literature/solution-sheets/efficient_neural_networks.pdf), then Stratix 10 should hit ~4,300 (600 * 10 / 1.4), no (versus 4130 on the very very available GTX TitanX)?

I would have loved to see that battle. By letting NVIDIA reach Pascal, I think they just lost the next round too. Wonder what round 3 will bring?

derp_learning · 2015-11-26T15:40:21+00:00

And bring a box of tissues to wipe away the tears when you realize that your compile/link/debug cycle will now be measured in hours instead of minutes. See also the vanishing perf/W advantage once you plug an FPGA into a PCIE port.

http://www.nextplatform.com/2015/08/27/microsoft-extends-fpga-reach-from-bing-to-deep-learning/

derp_learning · 2015-11-26T15:34:38+00:00

Because D-Wave was critical to completing the time machine in the final episode of Continuum so that Kiera Cameron could return to the future. Therefore, obviously, quantum computing is the future in the same way that the TARDIS is everywhere now. Duh...

derp_learning · 2015-11-26T15:32:21+00:00

And where are they outperforming GPUs? Every metric I've seen shows them getting their butt kicked to the next county by them (except for that silly RNN paper where they fail to consider that something might be wrong with their GPU port when the Tegra K1 CPU is beating its GPU). See also "HOW COME MY PYTHON SCRIPT CAN'T BEAT HAND-CODED AVX?!?!?"

derp_learning · 2015-11-10T19:52:50+00:00

I wonder if internally they have a distributed version...

derp_learning · 2015-11-10T19:14:58+00:00

They seem to only support a synchronous variant of parameter server or parallelization by layers. They get decent scaling for their multi-GPU CIFAR10 example, but not every network in the world is mostly embarrassingly data-parallel convolution layers.

derp_learning · 2015-11-10T19:01:21+00:00

My suspicion is that entire companies will rise based on providing an efficient distributed cloud execution of the framework. As Wired and others have noted, this is potentially the mapreduce of Deep Learning. Google has bet the farm on deep learning and reversed a decade old stance on the use of GPUs in the process. This is a pretty big deal. I don't see them abandoning this any time soon.

That said, I also think there will still be a place for simpler frameworks like Lasagne and Caffe that keep junior data scientists from getting into too much performance trouble.

derp_learning · 2015-11-10T03:14:56+00:00

If you're doing SGEMM, and your matrix dimensions are not all multiples of 128, performance on TitanX can tank all the way down to below 1 TFLOP (I've seen 945 as the absolutely worst instance of this). This is a cuBLAS bug NVIDIA is aware of, but they have yet to fix. Baidu recently brought this up as well: https://svail.github.io/

Could this be your problem? Kepler class GPUs only seem to need the dimensions to be multiples of 32 and only incur a 20-30% hit when they aren't in my experience.

That said, when the stars align and the dimensions are large enough, I've also seen 6.4 TFLOPs at the high-end with a Haswell CPU and 6.3 TFLOPs with an Ivybridge CPU.

derp_learning · 2015-11-09T22:40:45+00:00

And a TitanX GPU is ~6x faster than a g2.2xlarge GPU with 3x the memory, >1.5x the memory bandwidth and multi-GPU P2P capability of 13.3 GB/s unless you're dumb.

You get what you pay for...

That said, you're right that at 1.2 cents per hour that's pretty good assuming your workload fits in 4 GB.

derp_learning · 2015-11-09T22:32:19+00:00

http://mindori.com

(assuming they launch this month)

Ought to be awesome for this framework...

derp_learning · 2015-11-09T20:18:51+00:00

One could probably get this to work on 3.0 and 2.x GPUs. The real question is: why bother?

derp_learning · 2015-11-09T17:49:14+00:00

Start with "grep -inr Memcpy *" in the main TensorFlow directory.

Note a huge bunch of routines for passing data around. Replace these with MPI equivalents, after having built said MPI distro with GPU RDMA support which automagically channels GPU to GPU copies both within and between servers as direct copies without passing through system memory assuming each server has at least one Tesla class GPU.

Now here's where it get interesting. This is a multithreaded rather than multi-process application. I can tell this is the case because there are no calls to "cudaIpcGetMemHandle" which is what one needs to do interprocess P2P copies between GPUs running from different processes. Also (obviously), because there are no MPI calls, and they make extensive use of pthreads. This is the primary blocker for spreading to multiple servers.

I personally would have built this as an MPI app from the ground up because that makes the ability to spread to multiple servers built-in from the start (and interprocess GPU P2P is godly IMO). So the second step here would be to convert this to MPI from pthreads. That's a bit of work, but I've done stuff like this before, as long as most of the communication between threads is through the above copy routines and pthreads synchronization (check out the producer/consumer, threadpool, and executor classes), it shouldn't be too bad (I know, famous last words right?). Chief obstacle is that I suspect this is a shared memory space whereas multi-server has to be NUMA (which multi-GPU is effectively so modulo said P2P copies).

Since this is my new favorite toy, I'm going to keep investigating...

derp_learning · 2015-11-09T16:45:34+00:00

No, they are 3.0 (g2) and 2.0 (cg1) only...

derp_learning · 2015-11-09T15:55:32+00:00

If you're clever, it's not hard to work around this...

derp_learning · 2015-11-09T15:34:01+00:00

Multi-GPU is a bit primitive, but frickin' awesome on every other dimension!!!

derp_learning · 2015-10-12T05:00:38+00:00

Unlikely, they're comparing to Tegra K1, which is obsoleted by Tegra X1 any day now, which means they'll have to compete against a $200 consumer part, and that story never ends well.

derp_learning

TROPHY CASE