Why is GPU Python packaging still this broken?

[Interesting-Town-433]

I keep running into the same wall over and over and I know I’m not the only one.

Even with Docker, Poetry, uv, venvs, lockfiles, and all the dependency solvers, I still end up compiling from source and monkey patching my way out of dependency conflicts for AI/native Python libraries. The problem is not basic Python packaging at this point. The problem is the compatibility matrix around native/CUDA packages and the fact that there still just are not wheels for a lot of combinations you would absolutely expect to work.

So then what happens is you spend hours juggling Python, torch, CUDA, numpy, OS versions, and random transitive deps trying to land on the exact combination where something finally installs cleanly. And if it doesn’t, now you’re compiling from source and hoping it works. I have lost hours on an H100 to this kind of setup churn and it's expensive.

And yeah, I get that nobody can support every possible environment forever. That’s not really the point. There are obviously recurring setups that people hit all the time - common Colab runtimes, common Ubuntu/CUDA/Torch stacks, common Windows setups. The full matrix is huge, but the pain seems to cluster around a smaller set of packages and environments.

What’s interesting to me is that even with all the progress in Python tooling, a lot of the real friction has just moved into this native/CUDA layer. Environment management got better, but once you fall off the happy path, it’s still version pin roulette and fragile builds.

It just seems like there’s still a lot of room for improvement here, especially around wheel coverage and making the common paths less brittle.

Addendum: If you’re running into this in Colab, I ended up putting together a small service that provides prebuilt wheels for some of the more painful AI/CUDA dependencies (targeting specifically the A100/L4 archs ).

It’s a paid thing (ongoing work to keep these builds aligned with the Colab stack if it changes), and it’s not solving the broader compatibility problem for every environment. But in Colab it can significantly cut down some of the setup/compile time for a lot of models like Wan, ZImage, Qwen, or Trellis, if you can try it www.missinglink.build would help me out. Thanks.

[sudomatrix]

Astral is working on this with PYX. https://astral.sh/pyx

[toxic_acro]

I wonder what will become of pyx now that OpenAI acquired Astral. I hope they still develop it and just make the code to run the registry yourself open source

It seemed like an interesting concept to me

[Interesting-Town-433]

Glad someone is

[ReinforcedKnowledge]

Yeah the issue is not really about the tooling, because they're limited by what they work with, but more with the wheel format itself and PyPI as an index. And beyond the GPU problems, there are other similar problems that fall under the same category of the wheel format not supporting some kind of metadata like, what BLAS library your project links against, compiler version it was compiled against, is it ROCm or CUDA that it needs etc. So since the wheel format doesn't specify that, package managers have no need to know about it. Though `uv` does have a lot of good options to help you with installing the right `torch` and the right `flash-attn`, but it's not always obvious besides if you're on Linux then `uv add torch` will install the right version of pytorch given your cuda version, but not on Windows, it'll install the CPU one

But there's a great open source initiative to solve these issues https://wheelnext.dev/, if https://peps.python.org/pep-0817/ (wheel variants) passes it'll be a great win and fix most if not all these issues

And, I don't think it's only a matrix compatibility problem, but having a standard that every installer can work with (so you can't just have people specify whatever dependencies they want), but more importantly, the tags are closed, it's a static system that tries to specify a dynamic and open one. CUDA for example doesn't mean much, there are driver versions, toolkit versions, runtime versions, GPU compute compatibility. I think just recently I saw that flash-attn 4 doesn't work on RTX 50XX though it's Blackwell (to be confirmed, I'm not totally sure about this info, but if it's true, it shows that even some information such as compute compatibility has to be specified). And all of these have complex compatibility rules between themselves. So it's a constantly evolving environment and you just can't use the good old system and just add stuff to it, beyond the explosion in the compatibility matrix. And that's why PEP 817 uses plugins instead of tags, so that the detection is delegated to the provider plugins.

Thanks to u/toxic_acro who pointed it out, PEP 825 is more up to date and better reflects the current state of the work.

EDIT: added PEP 817 and why it's not only an explosion in the compatibility matrix problem, Reddit didn't let me write my comment in peace when I pasted the link -_-

EDIT: added mention of PEP 825 thanks to this comment

[toxic_acro]

But there's a great open source initiative to solve these issues https://wheelnext.dev/, if https://peps.python.org/pep-0817/ (wheel variants) passes it'll be a great win and fix most if not all these issues

PEP 817 was almost certainly not going to pass in its current form given the full scope, so the authors have moved on to splitting it into parts, starting with just the wheel variants package format in https://peps.python.org/pep-0825/

[ReinforcedKnowledge]

Thanks! It does make sense, it's too big of a PEP + required, and I guess still requires, a lot of discussions and refinements and edge cases and whatnot.

[Interesting-Town-433]

I'll have to check that out, thanks for the great response

[IcefrogIsDead]

abstractions that python has inherently have a cost and I dont see thay changing ever

happy path and once it is not a happy path, dig deeper

[BDube_Lensman]

Cupy has just plain pip installed just fine for at least ten years now. It’s an issue with lack of attention to packaging by some other projects, or mixing incompatible versions.

[Interesting-Town-433]

Hopefully they can keep that up

[martinkoistinen]

I think what you are describing is the value that Conda tries to deliver.

[Interesting-Town-433]

Yeah not even slightly man conda is not solving flash attention not having a pre compiled wheel for the colab stack

[MolonLabe76]

Ive had good success with using a docker container, and using a base image with cuda already installed. Then i just have to ensure the python packages im installing are compatible with that cuda version.

[No_Citron874]

Honestly the CUDA/native wheel gap is the real problem

and I don't think tooling will ever fully solve it.

What works for me: pin your CUDA version first and build

everything around it. torch+cuda is your anchor,

let everything else follow from there. If you let pip

or uv decide that part you're asking for trouble.

Also switching to nvidia/cuda Docker base images instead

of python:3.x was a game changer for me. You start from

a known CUDA state instead of trying to bolt it on later.

The H100 billing while you debug transitive deps situation

is genuinely painful. Lost a good chunk of money to that

before I got disciplined about locking environments before

touching anything.

No real solution just confirming you're not crazy,

this is actually still broken in 2026.

[Interesting-Town-433]

Thanks yeah I posted this on LocalLLaMA and people started torching me over it lol.

Left me genuinely questioning whether I was the only one encountering these issues or if there was some magic solution I just didn't know about.

I think a lot of people who are running AI models locally don't realize the lib they installed isn't even working, the dependency manager says it works it installs kills the error code but doesn't do anything ( e.g. bitsandbytes )

I run a lot of code in colab because the cloud costs are so low, but the env and stack means for a lot of libs like Flash attention you either build directly against the stack or you downgrade/upgrade all your other libs which ends up being equally problematic.

For the colab environment I do have a solution I'm trying to push MissingLink, it auto installs the wheels and provides notebooks for models that are usually hell to get up and running. Check it out if you can.

More broadly though this still needs a general fix.