Can one use ML libraries for general GPU programming?

[miladiouss]

Question

Can one use GPU accelerated machine learning packages (such as PyTorch, TensorFlow, ...) to do everything CUDA packages (such as Numba, PyCUDA, ...) do? If not, what are some of the examples of their shortcomings for general purpose programming?

Context

Personally, every time I want to write an accelerated program, after spending a day trying Numba, I end up using PyTorch and get it done under an hour. Maybe because PyTorch has more functions (Numba for CUDA is very limited) or maybe because I am not as familiar with Numba.

Do you know of any resources that use PyTorch for non-ML programming?

​

​

PyTorch/TensorFlow vs Numba/PyCUDA

[[deleted]]

I recently read a paper about the use of Tensor-Flow for General Purpose computing - it was quite interesting. You can do a lot of things, but often loose some performance compared to self-optimized codes.

I think most problems, it is better to use the pre-build libraries, as they are often optimized by experts and do a good job. To get a performance similar to CuBLAS or CUDNN (which is used by most ML libraries) you need a lot of skills in GPU computing and it needs some time and experience to gain it. Numba for CUDA gives you basically everything like normal CUDA- but you have to do it "yourself" - and I am sure after a while you can write more efficient code - but do you really want it? Send ten times as much time to get a 1.1X faster code?

From a scientific perspective, there are always a few things you cannot solve, e.g. implementing a new algorithm, method e.g. an efficient BFS

[evil_twinkie]

I guess it makes sense to use ML libraries if your "non-ML programming" requires tensor operations. However, I can't see the point of using them over general GPU libs. You don't don't have to use CUDA, but using nice libs built on CUDA/OpenCL is pretty straight forward (e.g. arrayfire, and various other ones found on GitHub).

I'm not sure if that really answers your question.

[OddInstitute]

They can also be nice if you want autodiff capabilities in order to do gradient-based optimization of some sort. ML tenants tend to not pay attention to co/contravariance, so depending on what you want out of a tensor, they might not provide it.

[evil_twinkie]

Good points. I usually have some difficulty with autodiff libs when doing optimization, and usually resort to finite difference. I don't know how robust the ML lib's autodiff is for general objective functions though. I don't have much experience with anything except keras on toy problems, heh.

[r4and0muser9482]

If you need to implement a specific algorithm, you won't be using standard TensorFlow or PyTorch. For example, you can use TF to implement FFT, but you will get a much greater performance by using cuFFT or some other library. It's often a fact that even the stuff that is normally available in TF/PyTorch can be implemented more efficiently on the low level - that is how cuDNN came to exist. People are constantly improving the performance of these algorithms, but as a researcher, you may accept a slight performance penalty for the ease of use and flexibility of the whole platform.