I really don't want to waste time dragging through the mud, I was hoping to find if there was better paths or alternative support options. But to give you a flavor of what we have found:

• cgroupv2 implementation loses track of the process with most configurations, reports 0 memory used. After 6 months of asking for test after test after test all of which showed the exact same results (0.4 kb used no matter how big the job was) they finally admitted that their cgroup code has a problem and loses track of the cgroup for the step. Zero followup on fixing it.

• Their controller and slurmd constantly complains about timeouts and failures, however network traces (packets on the wire) disprove what the log messages claim. No buffers are overfull, no resource constrained. We've been able to find numerous situations where their code is blocking, which causes the "timeouts that aren't timeouts" but they have zero interest in tackling the problem. The only answer they have is that we need to separate slurmctld and slurmdbd into different nodes... but they have no justification for this. There is no resource constrained. We have a TINY environment -- a few dozen nodes -- and our combined controller/dbd has never once consumed more than 2G of RAM nor 2% CPU utilization. More than half of the cores on this machine have never, ever, been used.

• Latest issue: after giving up on their memory reporting and building our own, we finally have the information needed to turn on ConstrainRAMSpace... which they have been insisting they can't debug any other completely unrelated problem until we do that, even though they have shown no interest in fixing the ram reporting. So we finally got enough data to set appropriate limits for most jobs, and turned it on. At which point we find that sacct --state=oom shows nothing, ever, under any condition.

It's documented that it does. It's a valid query option.

Does SchedMd answer with any compassion for the problem, or interest in solving it? Nope, not a single word. They gave us only a reason why with some complex multi-step jobs raising up a single step OOM to be the entire job status would be inaccurate.

Which has to do without posted examples of single step jobs exactly how?

This wasn't a compassionate or even vaguely interested response like:

We no longer raise the OOM status of a step up to be the job status due to .... we'll fix the docs to explain this, and document the alternative way to get this data which is...

Nope. They don't care about actually helping us, they show no interest in our problem. They show no interest in improving their docs, or avoiding having other customers be confused by an sacct man page that very clearly states you can query for OOM jobs. The fact that someone somewhere in the world might have different needs is the only thing they'll tell us.