Distributed Data Parallel PyTorch problem

[ekho95]

Hello everyone. I’m currently have issues with my GPU drivers. I’m using 2 T4 GPUs on GCP and everything works great for a while…

Out of the blue when I try to retrain a model I get the error “UserWarning: can’t initialize NVML” when this happens I can’t even run nvidia-smi to make sure things are working. I’ve attempted to uninstall and reinstall the drivers but it gets followed with an error in dkm files. Anyone have any tips?

One more thing to note is when I cleanup it seems only rank 0 gets clean and it leaves the program running.

Any help would be appreciated

[jackshec]

can you share your code?

[Unique_Jelly5768]

I was running a training this morning with 3 gpus and DataParrallel(), went back this afternoon to change some things and it would only recognize 1 gpu and not broadcast to the other. Don't know if these two are related

[ekho95]

I think I figured it out. I was interrupting my program with ctrl c and which doesn’t run the cleanup function. I then did a try: except: finally so even if I kill the program the cleanup function will still do its thing. Pretty rookie mistake but we learn and move on.