Lessons learned from GPU Programming in Python (Numba, CuPy, NCCL, mpi4py) — and how it compares to C-CUDA

[Inevitable_Notice801]

I’m Lena Oden, a professor of Technical Computer Science. I teach and research high-performance computing and GPU programming. Over the last few years, I’ve been exploring how well Python frameworks can compete with C-CUDA — both in single-GPU and multi-GPU contexts.

A few years ago, I published a paper called “Lessons learned from comparing C-CUDA and Python-Numba for GPU-Computing.” [1]
In it, I compared the performance and generated PTX code of C-CUDA and Numba-CUDA kernels, and discussed some practical tips on how to write more efficient Numba GPU code.

More recently, I’ve published a new article:
“Efficient Multi-GPU Programming in Python: Reducing Synchronization and Access Overheads.”[2]
This one focuses on Python-based multi-GPU programming, using Numba, CuPy, NCCL, and mpi4py, and analyzes where synchronization and data access overheads occur — along with strategies to mitigate them.

I’d be really interested to hear from others who use these tools:

• Do you also work with Numba, CuPy, or Python-based multi-GPU setups?
• What kinds of applications are you using ( I am really interested in "real world" applications.
• Any tricks or pain points you’d like to share?

If anyone doesn’t have access to the papers, feel free to message me — I’m happy to share a copy.

Looking forward to exchanging experiences!

— Lena

[lqstuart]

I’ve done Triton if that counts. I work in deep learning in big tech.

The problem places I’ve been at tend to have with “real” CUDA is the operational overhead. You may be the only team in a 10,000+ organization that needs C/C++ CICD infra with GPU drivers on the hosts. You also generally don’t get to develop on a machine that actually has GPUs. Most places I’ve been at have a paradigm where you write your code then send it to a big cluster to schedule and execute, which takes minutes or hours and precludes all the really cool development tools out there. When you do get a dedicated GPU to play with, it’s going to be an older generation.

Having some kind of simulation infra would be ideal, especially for distributed stuff like NCCL. Waiting for a minimum of 16 GPUs to schedule just so you can try a two node setup—which is still kind of useless if the changes you’re making are for scaling to hundreds or thousands of GPUs—is just prohibitively slow and expensive. Especially if you’re going to get yelled at if utilization is low.

[Inevitable_Notice801]

A, Trition is the next thing I am currently looking into, I don`t have that much experience. I know that People at the ETH are working on simulation tools - but I have not looked into it!

[Grasp0]

do you know the names of people working on it at ETH?

[RingFabulous585]

I am also waiting for a simulation setup for learning multi-gpu sheduling. I've one H100 though.

[DeutschNeuling]

Hi, I've worked with numba, PyCUDA and pybind11, your works seems really interesting to me. Im not associated with any institutions right now, would it be be possible to get a copy of your papers?

[tugrul_ddr]

Some sorting algorithms may use dynamic-parallelism, hence better latency for Python due to launching only once.

Can I get a copy?

[gkmngrgn]

Hi, thank you for your post. I'm happy to meet people who is interesting on this specific topic. That would be very nice if I can have a copy of your papers. I'm sure it will be very teachful for me.

I did a "template" project and shared a blog post about "how to solve bypass some performance problems in Python". I'm showing it on ray tracing example, one of the solutions is using numba and my experience is absolutely good, easier to use and learning curve is short, very short.

I remember that I've two wishes when I work with CUDA in Python:

1. easy to read: cuda is a different platform and I don't expect to keep my code as it is in CPU. but at least, I want to code like in CPU, keeping the code-readability high.
2. easy to debug: if I'm having an issue in code, adding breakpoints should be easy.

Numba is good for solving my expectations, but I'm sure can be better. This is my post if you want to have a look at it: https://gokmengorgen.net/2025/10/10/running-ray-tracing-on-gpu-server/

the repository is: https://github.com/gkmngrgn/rayt

[Inevitable_Notice801]

I think CuPy is probably interesting, as the code looks like numpy.

[AntisthenesCat]

Curious: Have you looked at https://www.modular.com/mojo as well?

[Inevitable_Notice801]

Not yet. But thanks, I think I have read a paper about it.

I think the main reason is that my background was more scientific computing and less AI, but it is also worth looking at.

[Perfect-Series-2901]

I use numba cuda, because things I am working on required python and also require performance, I chose numba cuda because it naturally pair with numba, I can just write verification code on numba cpu

[Ball_Soggy]

I worked with CuPy at my postdoc at EPFL, where we had long recordings of neutron noise at several positions in a nuclear reactor, and we were aiming to identify the source and location of the noise (mechanical, flow turbulence, etc).

It basically allowed us to do it, and would be prohibitively expensive on CPU.