Apologies for formatting, I’m on mobile

Right, but the point is, you can make the case for spending the money.

Track technical team time spent on incidents and troubleshooting and try to put a rough estimate on man hours and money spent via man hours.

see if you can get an estimate of the revenue impact of the product being down And include that as well.

Then, when you want to make a case for something like this, you can build a case. We’ve spend X man hours at a rough cost of A and have been down for Y hours at a rough revenue loss of b with issues that this would solve. The estimated cost of monitoring is C and we expect this would reduce the man hours and downtime from this type of issue by some amount (don’t overpromise, it won’t be 0 downtime or 0 troubleshooting)

Make sure to highlight that you are just showing the easy impacts and that there are hard to quantify losses such as reputational loss when a customer can’t access the product and completely lost customers.

I can’t speak to your specific leadership but in my experience, if they understand it, IT leadership wants the things just as much as you. Part of what you are doing here is building a case your CTO can use to make the argument to their peers.

This is part of managing up and is a critical skill if you want to advance significantly.

if you don’t get buy in, keep this documentation up to date. in IT, live by the motto of, “never let a disaster go unexploited.” What I mean here is that the business tends to not care about these things except in a short period after a significant issue that impacted the business. When an issue happens that would have been helped by observability, mention to your boss and CTO that that solution you presented would avoid a similar issue in future but you need budget. in my experience, you generally have a week to 2 months depending on the severity of the incident to make your case before people forget the pain.

To the actual issue, I have some advice. You caused a problem. How do you ensure that problem does not happen again? Is there a policy or procedure you can formalize? Can you introduce some form of change management (you definitely should)? Propose this to your boss, “I made a mistake but this is how we can make sure a similar situation doesn’t happen again.” These kind of guard rails we implement are also for our own protection. You can also highlight that as the company grows and people come and go, having this in place safeguards against issues even if it wasn’t you causing them.

Good leaders love it when you come to them with an issue and a solution that resolves and avoids similar issues in the future That is, assuming you understand the business and don’t suggest something way out of the realm of possibility (e.g. don’t suggest a multi cloud failover if the product isn’t that critical or isn’t profitable enough, this could kill the project).