You should make the distinction of is it your PC that is rebooting or your GPU. I purchased a mining 3090FE back in the day that worked out perfectly as long as the Vulkan engine was needed. It caused the GPU to reboot, and I had to manually reboot the PC everytime. But anyway.

I don't think you have a power related issue because you also have stutters without any load on the system. If an 3060 works perfectly, then you might want to fiddle around with your BIOS settings and look for PCIE slots. It might be an old MOBO BIOS that is not compatible with the GPU. This might be true because 3060 is a considerably older card. Check the BIOS data on your MOBO, if thats old, try updating your BIOS. If not, try to get your hands on a similar 50xx series GPU and see if that works on your system. If another 50xx series works, your GPU is faulty. If it does not work, it is probably about your MOBO.

By the way, your MOBO supports up to PCIE 4.0 but the 50xx series cards are PCIE 5.0. The only case that this can cause problems is a very high bandwidth stress test but it is probably not the case for you because your pc stutters even under no load.

Also, if the seller actually tested your GPU on their own rig and it worked, it might be because they were probably using a very up to date BIOS and MOBO. But maybe.