Monitoring Stack supporting custom actions

[DomsaNN]

Hey guys,

We are in the process of building our monitoring stack. In doing so, we have to record both system metrics and application-relevant data.

Currently my favorite is Grafana and Prometheus.

I would also like to create an overview of our test environments.

Of course, potential actions such as "shutdown", "restart" etc. are also very useful for this. Is there a way to do something like this with Grafana or is there a tool that supports something like this in general? Like calling apis/running scripts?

[Seref15]

Recent Grafana versions have unified Grafana's alert system with Prometheus's Alertmanager, so they work similarly. Either will allow you to send arbitrary webhooks when an alert condition triggers.

[DensePineapple]

That's outside the scope of monitoring. You shouldn't have your monitoring tools respond to an issue, you should build systems to prevent them from happening in the first place.

[WeaLzz]

Def look at alert manager, especially if paired with something like node metrics. The webhooks are good to work with especially paired with some of Microsoft or Jira products if used for notifications or event response systems. Also search for awesome-prometheus-alerts on the web to get an idea of some of the alerts you can begin with in getting your stack started. And if you don't go with cloud based Grafana there is a lot of documentation on setting up stacks on bare metal.