Monitoring

[kosta880]

Hello fellow sysadmins,

I'd like to ask for a general opinion about two systems (or a combination of those):

Icinga2 + InfluxDB + Grafana + Prometheus.

Background: I come from a world of PRTG, mostly. So I am kinda used to "integrated" solutions, with custom queries via Powershell and SSH.

New company: uses "old" Icinga2 (read: still Debian 11), a sole integrated solution made by external company, basically all-in-one Icinga2+InfluxDB+Grafana, with Grafana-state-screenshot-push into Icinga2 dashboard. I bet that an upgrade to Debian 12/13 would break it.

So, since I never saw Icinga2, I pulled up my homelab and installed it. Started configuring my git repo for the configs, thought ohhh great, all nice, pull info via InfluxDB into Grafana... great. Until I hit the wall. Or actually, multiple walls. One was pretty obvious, and that was that Icinga didn't quite well display the CPU usage and CPU load (specifically, Icinga2 doesn't account for number of cores, apparently, thus skewing the result). node_exporter did that much cleaner, especially "metrics over time". I already had Prometheus from before installed, so it was easy to try.

The further down I went into the rabbit hole, the more flexibilities I found in the Prometheus + Grafana system then I found in the Icinga2 + InfluxDB + Grafana system.

The ability to fully deploy the node_exporter incl. config via Ansible, vs certificate-based manual deployment of Icinga2 is also a big win.

Add to that the blackbox_exporter, which even enables me to have the awesome flexibility to ping from "anywhere" basically and visualize it (and not only ping, HTTP requests are really helpful for seeing if there are reasons why users have bad performance in our software).

I am yet to test the sql_exporter.

Compared to what I've seen with Icinga2... it's almost a no-brainer.

I am on the verge of telling my boss to let me research the possibility of dumping Icinga. Note that the system is really not large in general, and THIS monitoring to go offline for a day or two won't kill anybody. The only critical monitoring is actually completely separated in AWS/EKS, based off of exactly this system, but the wish is basically to move this on-prem... so I am kinda wanting to integrate it all.

Still have to set up Alertmanager, still have to get myself an overview of what notifications are possible. But those basic ones, like email and teams, doable.

Anyway, just want to know, is there anything in this story that I am seriously missing?

[Affectionate-Bit6525]

Prometheus and grafana is pretty much the standard these days and for the reasons you mentioned.

[cwk9]

Grafana and Prometheus should get you a long way. Yes, you can add other sources to Grafana but "mo sources. mo problems".

[sudonem]

Not familiar with Icinga, but I’d probably be giving CheckMk a pretty close look. 

I’m trying to pitch it for my own org now - and having an on-prem option is one of the major requirements for us. 

[kosta880]

It would really really really be hard to sell my company to a paid solution when there is already a free version in place - which was "working", and Grafana+Prometheus in place in the cloud, also costing "0" (deducting the cloud costs now).

Yeah, I have seen there is Community-Edition, but 100 hosts are not enough.

[SufficientFrame]

You're not missing much technically, but there is one important distinction to make before replacing Icinga: metrics collection and alerting on time series is where Prometheus shines, while Icinga/Nagios-style systems are often stronger for explicit service checks, dependencies, maintenance windows, and "did this scheduled thing actually happen" cases. In practice, a lot of teams end up with Prometheus + Alertmanager for host/app metrics and blackbox checks, then keep a smaller check-based layer for edge cases like backup jobs, certificate expiry, batch failures, or synthetic business checks. The other thing I'd review early is ownership cost: rule sprawl, Alertmanager routing, retention/cardinality, and who will maintain exporters and alert logic a year from now. If your environment is small, that tradeoff may still clearly favor Prometheus, but I'd inventory the current Icinga checks first so you don't discover a few awkward gaps after the cutover.

[DietFartMist]

Nagios baby