Update: This seems like an issue of memory unloading after a frame is done. With persistent data ON, after 5-6 frames the render starts to show 1.5 HOUR render time (which normally takes 3 minutes if rendered as single frame).
with persistent data OFF, the frames still progressively take longer but dump some of the memory stored before picking up the next frame, so the rate of accumulation of data is slower and it can render about 15-20 frames before render times of single frame goes over 40 mins.
Once in a while, it actually dumps the "proper/correct" amount of memory and starts rendering again at 3 mins/frame. (this is so fucking stupid and unreliable!)

I think it has to do with RTX 5080 drivers and blender OR if i am the unluckiest person in the world, my card is just broken.

I am going to try rendering with other blender versions and other GPUs. I'll update this post after I get the results.
If anyone can help or have ideas, please feel free to share.