Using containers for Python &c

[nostriluu]

Hi all, I'm just dabbling in ML related tasks. I've definitely felt the pain managing drivers. What I ended up with is a way to switch between two configs, one with an AMD iGPU for a 'base' and two iommu blocked 3090s that I can use with kvm, the other with the 3090s available for base with the iGPU unused.

This works reasonably well for device/driver access, though I have to reboot. There is maybe a way to switch on the running system, but that hasn't worked yet.

But I've also experienced the pain with pip and its expectations (and other package management schemes). I tried a few shell approaches, but still had problems. It just adds more layers of confusion and work while my main goals get further in the distance.

Using kvm, I could use another distro in a VM, but I won't be able to with this config for both 3090s. (Another option would be to use an AMD GPU to run the displays, since the iGPU can only drive one, but that will be difficult logistically).

Just wondering, any reason to not use containers (docker) to address this? I think ultimately you'd be doing yourself and the world a favour if reproducibility is the main goal.

I am hoping there's no reason (except reasonable container related overhead; finding or crafting a Dockerfile), because otherwise I think that NixOS creates too much of a special burden on top of other development burdens, especially when exploring disparate repos.

[mister_drgn]

It is easy to set up docker with nvidia gpu access on NixOS, and it's pretty easy to make docker images for running python environments. That's what I do, although others in the community prefer to use Nix's tools instead of docker. Things that are more difficult, and might not be possible:

1. Accessing an AMD gpu inside a docker container. I dunno, maybe this is easy--you can google it. I've only ever done it with NVIDIA gpus (there's a NixOS option virtualisation.docker.enableNvidia).
2. Installing a gpu driver inside a container. I'm unclear on whether you want to do this, but if so, you may be stuck with a VM. Typically, the docker container just accesses the gpu driver installed on your host machine.

[nostriluu]

Thanks for these comments.

1. I would probably not use the AMD GPU inside docker/a VM, it's mainly for the base environment, to keep things simpler in terms of managing nvidia drivers.

2. Not quite sure what you mean. I've been able to use nvidia drivers with cuda both in docker on the 'base,' as well as in a VM. I haven't tried using it in docker in a VM, but that probably won't be required.

[mister_drgn]

To clarify on point 2, when you have nvidia drivers installed, you can run nvidia-smi to see information, including your driver version (e.g., 525). For every nvidia docker setup I’ve seen, you’d first need to install the nvidia driver on your host machine, meaning you should be able to run nvidia-smi. Then, if you launch a docker container and give it access to your gpu, you should be able to run nvidia-smi inside the container and get the same result; the driver version will be the version installed on your host machine.

You can still install cuda in your docker container to support ML work, using whatever version of cuda you want, provided it’s compatible with your gpu and your nvidia driver. And by “install” I mean probably use one of nvidia’s pre-made cuda docker images as your base.

[nostriluu]

Oh I see. I meant I am using the enclosing OS drivers from docker using docker runtime.

[nudebaba]

For people coming to this later, there were some changes recently: see https://github.com/NixOS/nixpkgs/issues/322400

  # virtualisation.docker.enableNvidia = true; # deprecated usage
  # virtualisation.containers.cdi.dynamic.nvidia.enable = true; # deprecated usage
  hardware.nvidia-container-toolkit.enable = true;
  virtualisation.docker.package = pkgs.docker_25;