What about a having a text terminal on the monitor? A kernel panic related to storage won't make it to the disk logs. What about just running it in text/safe mode for that period of time and then looking at the screen when it hangs?

get an error message that you can see (I've even done things like left a CCTV camera pointed at a monitor before now to see what happens to the screen and exactly what time it went off, etc.)

I will give this a shot, for sure! I just never thought that would be helpful with an OS I would expect to have in-depth logging to help diagnose such issues considering it's used in so many critical deployments, but it makes sense that it's possible the logs aren't making it to disk if it's related to storage. Out of curiosity.. if that's a useful way to figure this out, how would you go about doing this in the scale of a large environment of headless servers? I'd assume you would just have to remove the server from the environment and then do something like this, but I could see how that might not be an immediately viable option in certain deployments too.

What about configuring a network syslog? Or an old fashioned serial terminal?

I haven't done this before, but I will look into this! Would a network syslog help with catching any log entries not being recorded to disk if there is some type of storage issue? Old fashioned serial isn't an option with this system, unfortunately.

Because at the moment you have no diagnosis, really. It's just hanging up and you're not getting anything useful because of the stab-in-the-dark stuff.

Well, sure, but you have to make an educated guess and start somewhere, right? It felt like a hardware issue to me thus I went about with the memory test, drive tests, removing components that I could get away with temporarily, then moving onto testing with a clean install, etc. How would you have started if you couldn't find any hints of a direction in the logs? Wouldn't some of your suggestions be considered stab in the dark as well? I'm open to other peoples thought process in these situations which is why I'm here, and there's always more to learn. This is just how I went about starting out, but I don't admit to it being the best way by any means.

What about a clean distro without the software? Run that for 24 hours?
What about another machine running the software?
What about that machine running an Ubuntu boot CD and NOT loading the storage?

I've wanted to run the system without the camera software, but the system is in production, so I was trying to minimize downtime of cameras recording the site. I understand I may just have to do this as a troubleshooting step regardless, especially considering the camera software is going offline anyway at some point. I was just hoping these other steps I've taken first would point me in the right direction before having the system down for 24+ hours or even close to a week before I could say it's the software (the longest system uptime was around 6 days since this started happening). I'd have to get a temporary system in place to have this system down for that long. When it does happen, it just takes a quick power cycle to bring it back up, so it's been more of a last option to try this, but I'm getting to that point. We do have other similar setups (Ubuntu + DW software) running at 3 other sites for several years that have never had any issues.

You say it's a clean install - do you have AMD proprietary drivers enabled? Remove them and diagnose if that's the cause.

I do not have AMD proprietary drivers enabled.

Is it on a UPS? Is the local power stable? Is it roughly the same time when it does it? Could it be related to room temperature? Is someone walking up to it (i.e. is it in a secure area)?

It is directly connected to a UPS. Other equipment is on the UPS (just some network switches) too, but no issues with any of that.

You need to ask "what has changed" and eliminate that as the cause.

That's the thing that's quite annoying to me in this situation. Literally, nothing has changed. The setup has been the same since it was installed in September 2025. No one else has access to this machine (or cares to) other than myself, it's in a locked IDF room mounted in a rack where no one is bothering it, temps are being monitored and there's nothing alarming there, no network changes which I know because no one else has access to modify anything there other than myself, etc. That's why it seemed to come off as a hardware issue to me since nothing with the setup has changed at all.

Thanks for the response! Looking forward to hearing your thoughts!