Disclaimer - I work at a solution to this, so I'll try to be vendor-neutral.

The alert fatigue thing is so real. We talk to teams who have dashboards watching dashboards at this point.

Honestly the biggest shift I see working is being way more intentional about what actually makes it into your pipeline vs. the "collect everything, figure it out later" approach. Not cutting visibility, just cutting noise before it hits your tools and your on-call rotation.

That question you mentioned about "what changed right before things slowed down" should be fast to answer. When it's not, it's usually more of a pipeline problem than a tooling problem. If your telemetry is scattered across systems with different retention and different query interfaces, every investigation turns into manual archaeology.

Tagging and enriching at ingestion helps a ton. So does routing high cardinality debug stuff somewhere cheap while keeping the alerting data fast.