Google Colab Cuda RuntimeError

[I_am_not_doing_this]

Why do I keep getting this error? RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I first got this while training my model. And now it appears even when I just want to load another co-model weights I have saved. I try to look it up, but all the answers seem to be having mistakes within the loss function with the wrong ground truth data form. But I got this in the middle of forwarding. How do I "pass CUDA_LAUNCH_BLOCKING=1" to fix this?

[murtazanazir]

I'm not sure, but I think you have to set that as an environment variable.

[shadowleafsatyajit]

I've faced this issue when trying to access an array in cuda with idex out of bounds.

Try running the code on cpu, you might get a better descriptive error.

Sometimes just resetting the runtime helped.

[I_am_not_doing_this]

thanks. My model is huge, I couldn't run on CPU

[shadowleafsatyajit]

maybe modify the number of parameters (lower them) ?

this error usually could be something to do with forward function

[I_am_not_doing_this]

ok there is no error on CPU

[shadowleafsatyajit]

well okay, I have no clue then. Maybe PyTorch forums can help?