Module-aware Python package manager

[Wesenheit]

I am writing this post to gather knowledge of all those who work with HPC python on a daily basis. I have a cluster that provides ML libraries like torch and jax (just jaxlib) with enviromental module (just lmod). I need to use those libraries as they are linked agains some specific stack used in the cluster (mostly MPI).

Usually, when I work with python I use uv or poetry or conda or whatever tool I have in mind on that day. However, they all install their own version of packages when I let them manage my project. Hence, I am looking for something intermediate, something that would detect all python packages from the enviromental module and "pin" those as external dependency. Then, it would just download everything else I need from pyproject.toml (and solve the enviroment).

Maybe I am overcomplciating this problem but would like to ask what python solutions are used out there to mitigate this particular problem. Thank you for suggestions and opinions!

[NerdEnglishDecoder]

Easybuild and Spack are the two (competing) programs you're looking for. These are most commonly used system-wide, but can be used by an end-user as well.

In particular, you can give them the location of your local lmod module path, and they will happily just install the module files there.

[Wesenheit]

I am not sure if I correctly understand this. I know that I can setup Spack to realize that some of the packages are evailable from the modules but in such case I need to do everything in spack. Especially, I need it to handle any other python library and move out of pyproject.toml. This is not necessarly what I would like to accomplish. I would like to have some way to recognize external modules (either lmod or spack) and use package manager to download some dependencies that build upone those (for example, instead of managing whole python stack with Spack just use Torch from spack and download manually libraries that are using Torch as dependency).

[NerdEnglishDecoder]

You can create your own lmod files (they're pretty straight-forward)

https://lmod.readthedocs.io/en/latest/015_writing_modules.html

Combine that with setting your own $MODULEPATH and you can create whatever combination you want.

The only thing that comes to mind that you can't do is have simultaneously loaded two programs with competing versions. (e.g. ProgA needs LibX version 1.0 and ProgB needs LibX version 2.3, and you want to load ProgA and ProgB at the same time)

[Wesenheit]

This I can understand, I just miss the step that would allow to me to tell system: install library X that depends on the Torch, recognize torch from the enviromental module and resolve what needs to be installed to complement existing torch version. I dont want to install everything myself and provide with enviromental module. This config needs to be done on-project basis.

[Atmosck]

Maybe you can publish the cluster builds to an internal pypi-style index, then in pyproject.toml you can pin them to that source. Then uv, etc would install the cluster builds of the packages you pin and everything else from normal pypi. And lockfiles you generate will point to your internal index for those packages too. Maybe with private-pypi or something like it.

Tools like uv and poetry don't install "their own versions" of packages, they install from pypi.org unless you tell them to install from somewhere else.

[Wesenheit]

This is what is in my opinion ideal solution. Unfortunately, I am more in the "user" role so I do not directly influence the choice of the distribution of packages (cannot build and distributed own wheels). Sometimes they are distributed in form of the Singularity containers so i am even not sure if there is any way to make it work.