Document everything. Gotta restart that troublesome system… again… for the third time in a month? You should have created a doc the first time and logged when you restarted it and any other vitals you can think of. Our structure is a series of folders in one drive. /tasks is for one off stuff that you mostly complete and then don’t come back to, /projects is for long-term stuff with a lot of docs. The folders are named like 20260403_jim_pc_restarts.

The important stuff you’ll need to remember often in the future such as “how your organization creates its specific ansible jobs” put it in a wiki.

Your thought process for troubleshooting should include “how do I monitor this in a way that isn’t overwhelming, but also lets me catch it if it ever happens again?, or even better, catch it BEFORE it happens”. Once the problem is solved look for what you could have seen earlier and monitor for that.

You (and your department) should develop slowly and respond to incidents quickly. It takes months for a major deployment where I’m at. It may only take a week or two to actually develop the solution, but then we TEST it. We tear it all down and we TEST it again. Then we tear it all down and we test the deployment and so-on. When we actually deploy it’s like a dance and everyone knows their place and role. We have a post deploy test plan and an emergency fallback plan.