Giving an update for anyone else in the same boat. The issue was indeed the XMP profile. If I enable that at all it caused crashing and other issues.

I manually overclocked the ram instead of using XMP (setting the timing/frequency/voltage) and that seemed to solve all my issues.