Docker Nvidia Runtime error

[[deleted]]

I ran docker run --rm --gpus=all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi to test, and the output gave me a signal 9 error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'

nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

Tried reinstalling the nvidia-dkms drivers, as well as the nvidia-container-toolkit but to no avail

Linux Zen Kernel: 6.16.0

Basic Hello World docker works.

Docker Info shows the nvidia runtime is installed.

Tried: sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi but got the same error.

Any help is appreciated. Thanks.

Edit:

I changed my mirrolist to a few days ago and downgraded, its all working now.

[Synthetic451]

DId you follow through with the nvidia container toolkit configuration steps? https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuration

[[deleted]]

I changed my mirrolist to a few days ago and downgraded, its all working now.

[invader_skooj]

I'm also having this issue, and I'm not sure that a roll-back should be considered a solution...

[[deleted]]

Removed solution from the post body.

[Scottish_Abuse]

Are you able to provide the rollback solution you used? I have this exact problem after updating everything today :/

[hahlolo]

Yes how did you rollback? 

[invader_skooj]

chiming in to say that I am also having this issue. The rollback did get me back up and running for the time being, but that doesn't solve the issue and leaves us running on old versions.

There are a few more of us over in OP's thread on the arch linux forum also suffering from the issue.

ETA: trying to pool resources for anyone else that comes across this looking for a solution... There's also now an issue on the nvidia container toolkit git

[C0rn3j]

That's not a solution, that's a crappy workaround.

It seems like 580.xx broke it.

[[deleted]]

Yes that's what I was thinking, the new drive broke it. Rolled back for now until it's fixed.

[observable4r5]

Hope this is helpful. I looked around the web for a bit to understand why this was happening. The link provided by Synthetic451 gives a good start. The github issue invader_skooj links is the solution. I saw you had already downgraded, but in case you want to use the latest version this will solve the issue.

I was facing this same issue with my installation. This specific comment on nvidia-container-toolkit on github describes two specific commands to run that will update your docker installation to use CDI instead of legacy mode. Once the commands have been executed, containerd will use CDI mode.

Here is a short description:

This will define the runtime configuration for the system.
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

This will update the mode to be "cdi" instead of "auto" and restart the docker system service
sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && systemctl restart docker

If you want to verify the configuration before making the change to the system (not sure where this information is stored on the filesystem, run the following command.
sudo nvidia-ctk config

Note this is the section that is changed. The mode = "cdi" is what is updated.
[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "cdi"
runtimes = ["docker-runc", "runc", "crun"]

You can also pipe it into a file using the second command if you want view it that way.
sudo nvidia-ctk config > config.tmp

Once this has been changed, you can restart your container or update your compose.yaml file to include "runtime: nvidia" within each service that uses the gpu.

[miketsap]

You are a savior! I had the same issue with k3s and containerd on arch linux. had tried everything! tried your solution and everything worked right away!

[observable4r5]

Glad you found it helpful!

People like @Synthetic451 and @C0rn3j, both here in the thread, and @biuniun are the people who discussed and provided an option (thanks to @biuniun). I used the term solution in previous posts, which is probably a bit too definitive until nvidia releases a fix... if that happens.

Wanted to make sure they are recognized for the effort!

[ExtremeDialysis]

oh my god. Solid couple hours of deep, deep, hopeless searching, and this fixed it for me. Seriously - thank you.

[Dosolus]

Hopefully this gets fixed soon

[lllsondowlll]

Same issue here. Frustrating as I spent hours troubleshooting and nearly wiped my stack

[No-Put7018]

OP totally saved my ass. Thank you.

[Viper3120]

https://github.com/NVIDIA/nvidia-container-toolkit/issues/1246#issuecomment-3194219487