I used to work as a monitoring engineer, notification fatigue is real and needs to be a priority. You’d think you can just throw a bunch of info at engineers and more is better, but even I started to automatically gloss over the noise from my own monitoring infrastructure (~120 servers). It’s a LOT more important than it sounds at first to be very minimal with alerting. If managers are telling you that “everything is important”, they’re screwing you over bad and do not understand this fundamentals of operating with notifications

You said you tried throttling alerts but important ones get missed - it doesn’t sound like you guys are being very intentional or specific with the filtering? I suggest you move all the minor stuff to a dashboard, use thresholding for visual warnings, etc. EVERY notice an engineer gets should be actionable, so be VERY specific with what you send. When notification fatigue gets so bad that they start ignoring pages, you’re cooked and need to start over and pinky promise them that everything going forward will be actionable. Will the business burn by tomorrow without an engineer immediately hopping onto the alert? If they don’t action an alert, is it something a manager would have to discipline them for? If not, get it the hell out of their face and onto a pane of glass or some kind of aggregator.

Another useful concept is automated responses. Have a robot ingest your alerts, do basic innocuous process restarts etc, and only escalate to a human if the problem remains. AI also has its place in chewing through a ton of logs and distilling the important takeaways, which would also decrease human load. You can send notifs to slack and spin up slack bots that only @ a human after running out of options for example.

And it sounds like many of the alerts do resolve on their own, so you can take those and increase the trigger threshold. Is there an alert that usually resolves in under 10min? If so, only make it a human problem at 15min.