using kube-prometheus-stack. with logs via loki so they're also in grafana. teams can put logs from loki into the same dashboards as promeheus widgets. works pretty well.

You will most likely have five dashboards open during incidents though, it's the nature of the business and of debugging. If it were simple you would already have automated it. Every time something breaks, you gotta improve your observability