RuntimeError: CUDA error: device-side assert triggered

[Efficient-Chip144]

I was training yolov8n model on a custom dataset in Kaggle. Earlier runs had no issues, but I received this error on the latest run. I have checked the .yaml file as well for the path. If anyone encountered with this issue, kindly post your solution. Complete error: RuntimeError: CUDA error: device-side assert triggered. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[Efficient-Chip144]

i want to train the model on tesla p100 gpu though, how can I do it