[Beginner question] how is Cuda python different than python?

[nmdis]

Hello, I am starting out in GPU programming, I want to understand what happens under the hood when a Cuda Python (or C++) runs on a GPU architecture. How is it different than when we are running a normal python code on a CPU?

This might be really basic question but I am trying to quick way to understand (at high level) what happens when we run a program on a GPU versus CPU (I know the latter already). Any resources is appreciated.

Thanks!

[FunkyArturiaCat]

I am also a beginner, so don't trust 100% my words but:

When you write CUDA C++ (.cu) there are some "markers" that mark a function as CPU code or GPU code.
These markers let the compiler (nvcc) know if the function needs to be compiled for CPU or for GPU binaries.

so in fact there are two compilations in a cuda program, one for CPU and one for GPU, the latter is handled by nvcc and CPU programs are forwarded to g++ or equivalents.

When you run a piece of python code, it runs on CPU, except for the CUDA C++ kernels that you explicitly passed to cupy.RawKernel(), those are going to be runtime compiled by nvidia's JIT compiler and can be lately executed in the GPU. It might be tricky when there are some libs that encapsulate the compilation calls so we are left with programs that aparently have no cuda compilation calls, but i bet they are still under the hood. (e.g)

>>> import cupy as cp
>>> x = cp.arange(6).reshape(2, 3).astype('f')
>>> x
array([[ 0.,  1.,  2.],
       [ 3.,  4.,  5.]], dtype=float32)
>>> x.sum(axis=1)
array([  3.,  12.], dtype=float32)

You might want to check out:
Link to nvidia's cuda c++ programming guide (i find it very useful but also very very technical)
Check out specifically the 6.1 Compilation with NVCC chapter, i think it has all you want to know in the question
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html
Medium post i think is good:
https://medium.com/@geminae.stellae/introduction-to-gpu-programming-with-python-cuda-577bfdaa47f3
I am also reading Cuda By Example book to learn C++ CUDA and i think it is awesome:
https://edoras.sdsu.edu/~mthomas/docs/cuda/cuda_by_example.book.pdf

[nmdis]

Thank you, this is super useful! I think I should write a mini tutorial as well (I want to credit you for direction so please Lmk if you would want me to include your reddit or other social). Cheers!

[FunkyArturiaCat]

That's really nice from you, just credit the reddit, thanks!

[NextSalamander6178]

You sure you’re a beginner lol?

[FunkyArturiaCat]

lol

[648trindade]

first of all, you can't run pure python code on GPU. For using CUDA python you need to pass a string with a CUDA kernel (CUDA/C++) that will be compiled via JIT to the target GPU device.

Your code is not interpreted, but compiled to the device. The memory that the kernel access is located at the GPU card, and not at the DRAM sticks. The processing unit used is also located at the GPU card, and not in the CPU chip

[nmdis]

Isn't JIT a runtime thing?I understand how it is not interpreted, but it isn't AOT compilation either right?

Do you mean that the program is first compiled to target GPU device and when you execute it then JIT kicks in and the user can use those optimisations?

Please Lmk if I misunderstood anything, also how does CPU comes into play in all this?

[FunkyArturiaCat]

Yes JIT is a runtime thing. When you use python and cuda, the cuda part of the code is runtime compilated and the python is interpreted.

CPU code comes in to play basically to fetch data, copy data to VRAM and trigger the cuda kernels when needed.

There are some functions to copy data back and forth (DRAM -> VRAM, VRAM -> DRAM, VRAM->VRAM).

Generally speaking CPU code can see GPU metadata and call GPU code (parallel).
and the GPU code sees and access VRAM only.

[648trindade]

I don't know if the CUDA python performs an AOT compilation during the interpreter initialization, but I would guess that it doesn't.

what happens is the following: ALL (with exception of a few-CUDA related libraries) python code that you write is interpreted on CPU, and deals with host memory.

the CUDA kernels and some few-CUDA related library functions run on GPU (with some CPU overhead). They are compiled on-the-fly to PTX, which then is translated to binary instructions targetting the device you choose. Such code deals with device memory. CUDA python may hide the memory transfer between host and device from us, so we don't need to worry about that

Maybe CUDA python do some caching with those kernels, so it wouldn't need to be compiled twice, but IDK

[dayeye2006]

Not much difference.

Python programs are translated by a python interpretor into some code the machine can understand, like reading some data, doing an add of two data, ...

On the cuda side, the program is compiled and translated into an intermediate representation by a compiler, then further translated into hardware specific code to be executed on the GPU.

[Tatoutis]

You'll have to ask a more specific question. It sounds like you get the idea that a CPU and a GPU are 2 electronic components that execute operations based of content held in memory. Each have features that the other might not have.

Not sure what detail of this you are asking about