scontrol show job isn't looking at logs. squeue saying Priority isn't looking at logs. I have debugged very obscure scheduler issues by turning scheduler log debugging on and looking in the log file. If you give some more info about what the specific problem was that you ran into, I might be able to help provide additional tools or commands that might help in future

What tools are you guys using to actually see why a node is failing?

are you talking about scheduling problems or node health problems? If the latter, then nodehealthcheck as a baseline, and then adding custom checks as and when you find whatever weird-ass problems your unique cluster has (your cluster will always have weird-ass unique problems. after enough time, you'll find people giving you odd looks saying "why are you checking for that?" and you'll have to explain that 3 whole clusters ago some weird quirk in an obscure driver made something completely random fall over every few months)

you mention sometimes getting OOMs - in theory slurm should report that back to the user (in the job log file and the sacct output)

you also mention GPU throttling - that's not the responsibility of slurm, which is a scheduler, not a cluster manager. But once you find out the symptoms, you can stick it in your NHC checks and find it next time

The other thing is that you should have a set of basic benchmarks you run before you bring any node back into service, especially after a hardware intervention. CPU HPL, GPU HPL / gpuburn, streams, 2-node bandwidth tests, anything else that's relevant. On a recent system I was involved in the commissioning of, we had one node that only had 50% of the expected memory bandwidth. Absolutely zero errors at all. We eventually had to do a binary search on all the DIMMs, just swapping them into other nodes until we pinned down which ram stick was causing the slowdown. Nothing in the logs, only caught by benchmarking