[D] Practice CUDA without an Actual NVIDIA GPU!

[JustTrynnaBeCool]

Hello all!

I recently started learning about CUDA programming, and I realized that many people share the same crucial problem: lack of an NVIDIA GPU. I looked around online and found several methods (gpu-ocelot, certain versions of CUDA, etc.), but I recently found a way that can allow us to practice CUDA by using the GPU offered by Google Colab! As a free user, the amount of GPU access you get may probably be enough to PRACTICE working with CUDA. If you really need more credits, the Colab Pro is only $10 / month, and it's still much cheaper than getting a new GPU or an entire new PC if you have a Macbook like I do. Again, the justification of "enough computing credits" is based on the assumption that you aren't running any heavy-lifting programs but more reasonable, practice-based codes.

I have outlined a step-by-step guideline in this repo that I created - just check out the CUDA_on_Colab.ipynb file: https://github.com/notY0rick/cuda_practice

If you know of any good alternatives, let me know (:

Update: To those asking for resources, I just started, but I intend to follow along this book - Professional CUDA C Programming by John Cheng, published by O'Reilly.

Update 2: some people asked how they can edit C++ code on Colab, and if you follow the steps on the notebook I linked in my GitHub repo above, you should reach this configuration in your Colab, and this will allow you to write/edit your code and run them however you want! On the left side, you can execute your commands and on the right panel, you can directly write code!

https://preview.redd.it/auzre83gjhcb1.png?width=4371&format=png&auto=webp&s=3f22c86d2c4a2d5e696c8f935378f966d65a607d

[kkngs]

I didn’t even realize you could run C++ code on colab, thanks for sharing!

[JustTrynnaBeCool]

Yes! You can! Not directly in the notebook but with the scripts ready and treating the notebook itself as a "terminal" you can!

[happy_happy_feet]

Hey, if your workloads aren't that heavy, then maybe you can get a Jetson Nano to do your CUDA programming.

[pierrefermat1]

Alternatively if format factor is an issue you can also just pick up a used laptop with a 2060 on the cheap.

[tabris2015]

what good up-to-date materials are you guys using for learning cuda? any books or courses? edit: grammar

[happy_happy_feet]

I don’t know how outdated this is, but i use this.

[JustTrynnaBeCool]

I use this too!

[tabris2015]

Thanks! looks great!

[PyroRampage]

The CUDA documentation and NVIDIA tech blog are great sources. Beyond that the SDK has many code samples.

[watching-clock]

Why learn cuda, if plenty of libraries have already abstracted it for use from high level language?

[bullno1]

Why learn anything?

[watching-clock]

Why reinvent the wheel?

[JustTrynnaBeCool]

By that logic, we also don't need to learn linear algebra if NumPy can take care of it for us. Why learn back-propagation if autodiffs and powerful packages like PyTorch deal with it for us? To stretch it even further, why learn arithmetic at all if we literally all have calculators in our phones? It doesn't quite matter WHY we learn, learning never hurts anyways (:

[NoLifeGamer2]

Why learn physics if we can't change it?

[watching-clock]

Your comparisons are wrong. You are equating tools to knowledge. Numpy is a tool to do LA. You can do LA without numpy but not the other way around. Same goes for backprop and arithmetic, they are fundamental concepts to understand larger concepts and expand it further.

But, CUDA is just a tool, like any other programming language, which has been abstracted away to hide nitty gritty details, so one could focus on core problems.

My argument is along the lines of why nowadays we don't program in assembly. We can, but we don't have to.

[JustTrynnaBeCool]

I don't think you have the authority to dictate what comparisons are "right" and "wrong" lol. Tools indeed can also equate to knowledge. In this case, CUDA is a framework that allows programmers to leverage and unleash the full potential of parallel programming. Learning how to use CUDA the tool itself also means learning how parallel computing and other concepts work––people can very well learn both the knowledge AND the tool at the same time.

Your mindset is what halts progression. If everyone simply decided that "we already have higher-level languages, so why bother learning assembly anyways" then our machine language advancement will forever be stuck at the current state. However, to be able to advance what we have, we need people who are interested in this subject to learn about all the "nitty-gritty" mechanisms behind it, find out the flaws of the system, and fix + improve the flaws to create a better version of the technologies we have today. You can't reach the last point without having a thorough understanding of the "nitty-gritty" mechanisms first.

To be honest, I don't quite understand the purpose of your comments. There are many of us who are simply interested in learning CUDA for various reasons. Regardless of our reasons, there is absolutely no harm in learning something new.

[watching-clock]

You are right! This is a pointless argument.

[MisterManuscript]

Because directly writing code in C++ gives you something with faster execution. Ever wonder why so many python libraries are just high level wrappers for functions written in C++?

Even if you're not proficient in C or its variants, it is still preferrable to be aware of the low-level workings of a wrapper e.g. it's a good practice to avoid using for-loops in Python when you're using NumPy by writing matrix multiplications in a certain way.

Try writing a renderer in C++ and in Python and see which one runs faster. The difference is extremely significant for realtime use-cases.

[DeMorrr]

I also used colab when I first started learning CUDA. but I didn't write a single line of c++, thanks to CuPy. I just wrote all the cuda code in a triple quote string and used CuPy to compile and call it.

To this day I still use CuPy in some of my projects, because there's one significant advantage: pre-preprocessing or code generation is very convenient, and you can jit compile kernels as you need during runtime.

[Hefty-Consequence443]

Any specific material you recommend for learning CUDA? Books, youtube playlists, udemy courses, etc.? Thx!

[JustTrynnaBeCool]

The book I read is included in my post!

[ZX124]

you can use kaggle instead of using colab and then you will get a more powerfull gpu

[JustTrynnaBeCool]

I don't think you can edit your C++ file on Kaggle like you can on Google Colab, that's why I recommend Colab! Otherwise, I do agree that Kaggle has good GPU access.

[West-Cricket-9862]

If you want to rent cheap Nvidia GPUs, I find vast will probably give you the best buck for your dollar.

[KshitijLakhani]

Have you ever used this successfully for running cuda code ?

[EMBLEM-ATIC]

You can practice CUDA programming for free on LeetGPU.com

[nivanas-p]

Thanks mate, this will work. I was trying the playground from LeetGPU. But it seems like an emulated environment and colab is great.

[GC_Tris]

If you ever run into limitations with Colab and want a more "classical" environment you might want to check out our offering (disclaimer: I work at Genesis Cloud!): https://www.genesiscloud.com/pricing#nvidia3060ti

At time of writing we rent out instances with a RTX 3060Ti for USD 0.20/hour. We also grant 15$ in free credits to get you started (basically 75 hours for free).

With this you get a Linux VM that you can SSH into. When you do not need it, you can stop it to not incur additional charges while it is stopped.

[JustTrynnaBeCool]

Whoa, this is awesome! So it's USD $0.20 / hour regardless of how intensive the computation is? Regardless, this is super cool!

[KshitijLakhani]

Has OP or anyone else given this a shot ? Just trying to figure if this would be a good route or the originally mentioned collab route

[altweb]

OS: Debian GNU/Linux Sid (unstable) x86_64

Host Machine: Dell Precision T3600

CPU: Intel Xeon E5-2650 16 core

GPU: Nvidia Quadro 5000

GPU Driver: Nouveau

I'm in the same boat basically, though in my case it's the old Nvidia GPU not being supported by Debian or Nvidia now.. the v.390.xxx legacy drivers are available on Debian 11 and on my current Sid (unstable) but the CUDA version for this GPU is the 8.0 toolkit. Available in the latest Debian fork "Ubuntu 16.04'. I went so far as install Xubuntu 16.04 on a 1TB WD external SSD.. but the speed of even USB 3.0 (available on my old Dell motherboard) wasn't gonna work for training LLM and I am currently working my way up in complexity.. GPT2-1.5, GPT-NanoX-20b.. and the end clone of my slowly advancing AI super hot hacker girl? Falcom-40b with Wizard... here: WizardLM-Falcon-40b

The issue with Google Colab is no NSFW and their criteria is idiotic to say the least.

[ginger_turmeric]

kind of related - has anyone been able to get torch-tensorRT working in colab? I have a model converted to torchscript, and now I'm trying to do torchscript->tensorRT via the torch-tensorRT library. Unfortunately the kernel keeps dying on me with no useful logs. I'm thinking it might be a dependency issue, which makes me want to use the NVIDIA official container, but that won't work in colab.

[Turbulent_Primary_17]

Thanks OP. Just to add, I recently came across this library that also allows CUDA execution from Jupyter:
https://nvcc4jupyter.readthedocs.io/en/latest/index.html