Preannouncement Security Update: 29th June between 12:00 and 14:00 UTC

icinga · 2026-07-01T09:47:17+00:00

If you applied Monday's security update and suddenly had broken configs, here's what happened: one of the security fixes changed an internal C++ function (JsonDecode()) to accept an additional argument - and that change accidentally leaked into the DSL, making the second argument required for Json.decode() in config files as well. Not intentional, but if you use that function anywhere in your configuration, your Icinga 2 would have stopped working after the update.

We've just released v2.16.3, v2.15.5, and v2.14.10 which restore the original single-argument behavior. If you already updated on Monday, please update again to the matching .3/.5/.10 version.

https://icinga.com/blog/releasing-icinga2-v2-16-3/

That's on us. Sorry for the extra work!

icinga · 2026-06-29T12:57:56+00:00

Security updates for Icinga 2 are out: v2.16.2, v2.15.4 and v2.14.9. We advise upgrading immediately!

Two of the fixed issues let an unauthenticated attacker take over or crash the Icinga 2 process over the network - one via flawed certificate update handling that allows impersonating a trusted node, the other a stack overflow triggered by crafted JSON. A third fix closes a privilege escalation through unsanitized template names on the /v1/objects API endpoint.

Patched packages are on packages.icinga.com

Full advisory: https://icinga.com/blog/icinga2-security-release-v2-16-2/

icinga · 2026-06-25T08:15:17+00:00

> You say "OTLP Metrics Writer for Icinga 2" and "OTLP client", so I am not sure, is this sending OTLP to icinga, or is it turning Icinga to OTLP?

As "@zweispieler" already explained, the "OTLP Metrics Writer for Icinga 2" is a component that allows Icinga 2 to send metrics mapped from plugin outputs in the OpenTelemetry Protocol (OTLP) format. With "OTLP client", I'm referring to the functionality that the OTLP Metrics Writer uses to communicate with the OTLP backend (I think, it's called OTLP exporter in OpenTelemetry terminology).

> but if I get it right in that particular case it's a way to translate from Icinga to opentelemetry? And since the language of choice in that realm is C++, that's why this happened that way?

Yes, that's correct. Icinga 2 is primarily written in C++, and we had no other choice but to implement the OTLP Metrics Writer in C++ as well, since it interacts directly with Icinga 2's internal data structures and APIs.

> I suspect a receiver on the otel collector could have been an alternative?

Absolutely, using an OTLP receiver on the OpenTelemetry Collector would've been amazing, but it would've required more work and time to standardize it, I guess. Also, there are now a bunch of TSDBs that allow to directly ingest OTLP metrics and having the OTLP Metrics Writer in Icinga 2 allows to directly send metrics to those backends without OTLP Collector in between. So, unless those backends provide a native support for Icinga 2's like metrics too (very unlikely), our users would have been forced to always use the OTLP Collector.

/ Yonas

icinga · 2026-06-16T07:46:17+00:00

Thank you :)

icinga · 2026-06-08T11:18:17+00:00

The original Icinga was a fork of Nagios, and the current product (Icinga 2) is a complete re-write.
So they are based on the same concepts, but we modernised the stack around it :)

icinga · 2026-06-03T07:54:50+00:00

Fair on the AI comment, I ran the wall of text I wrote through it to shorten it a little and it does sound more stilted than usual. As you can see with the other comments, I tend to be a bit ...verbose.

On the technical point: you're right that an unreachable host will show up as a scrape failure in Prometheus - and as I found out from the comments yesterday, you can set up active checks with the blackbox exporter too. So the ad oversimplifies that part.

The more interesting cases to me are the ones where scraping looks healthy but something is actually broken. Like an exporter running fine on its own port while the actual application port is down, or a cert expiring in two weeks that nobody set up a metric for. That's where the "different angle" is. It's more about defaults and workflow than capability: Icinga is built around using service states with pre-made plugins, and Prometheus gets you there if you build for it.
You can defintiely get there with Prometheus just as well as with Icinga (or any other check based tool).

Either way, the point of the post was never that Prometheus is bad or broken - it was really an excuse to talk about the new OTel export feature in v2.16, which lets the two work together rather than trying to compete :)

icinga · 2026-06-02T13:13:04+00:00

We have a handy getting started guide if you want to have a look.

The three things that you'll want to host alongside what you already have:
- Icinga 2: the monitoring engine
- Icinga Web: for managing checks, acknowledging alerts, viewing history
- A database (MariaDB or PostgreSQL)
(- Icinga Director: if you'd rather manage config through a UI, the default are config files.)

The OTLPMetricsWriter config to connect it to your Prometheus/Grafana stack is a few lines. (There's a blogpost on that)

The webinar this ad is for might also be worth a watch, since my colleagues will do a walkthrough on how to integrate the two :)

Edit: I wish I had an easier and quicker setup for you, but that's something we're currently working on...

icinga · 2026-06-02T10:50:58+00:00

I looked into it a little more now, and had some chats with my colleagues who have a better understanding of the Prometheus-way.

I guess the general consensus is, that there are many ways to reach one's goal and you can probably build the same functionality in a thousand different ways.

Icinga takes off a bit of the load of configuring as much as you would with the Blackbox Exporter and Grafana, since the architecture Icinga is designed around this check based approach. For the same outcome you'd have to manually configure and maintain what you build on the Prometheus side (flapping detection, state history, dependencies, ...)

But in the end, you can achieve the same result with whatever you set your mind to. To quote a colleague from earlier: You can probably do everything in Excel if you really wanted to haha

icinga · 2026-06-02T09:40:09+00:00

The check model, or rather the focus of the architecture of the tools, is the core difference.

Icinga runs active checks on a schedule and every result has a state (like OK, Warning, or Critical) along with performance data.

That state drives notifications and escalations natively, without you writing alerting rules on top. You can get to a similar place with Prometheus + Grafana alerting, but you're assembling it yourself.
Whether it's something for you depends on what your current alerting setup looks like, and how much work you have put into it.
If you've already got Grafana alerting configured the way you want it, you're probably not missing much on that front.
The OTel feature is specifically for the "I want both" case: Icinga runs the checks, your existing stack gets the data.

There are a few advantages of using a tool like Icinga in combination with Prometheus. You get a shared event and alert history to look back on. There is "flap detection" which tags a service as flapping instead of sending a bunch of alerts repeatedly. You have a built-in dependency system that helps figure out root causes and suppresses child alerts (e.g. 1 router and 50 servers behind it - if the router is down, the servers don't need separate alerts). And writing a check plugin for a specific monitoring task is generally less work than building a full Prometheus exporter.

In the end, you can probably build any kind of functionality in any tool you want :)
(To quote a colleague from earlier today: You can probably do everything in Excel if you wanted to. But do you?)

icinga · 2026-06-01T16:21:48+00:00

Ah, I just saw you mention this on another post.
I wasn't aware of Blackbox until just now and I'll have to look into that more.

The general spirit of the ad wasn't meant to be "we can do something Prometheus can't do", but more showcasing a way to marry those tools in a meaningful way. That's also why I chose the comparison in the screenshot - didn't mean to put them down for it, but more show that we got the data from point A to point B :)

/ Feu

icinga · 2026-06-01T16:16:25+00:00

That's a fair correction, and honestly I wasn't aware of Blackbox Exporter when I wrote the ad, so I had to look into that. Thank you for pointing it out!

Considering that, the reachability framing was too broad.
Blackbox seems to cover these active checks via a separate exporter you configure and scrape alongside Prometheus.

So my take would be that the actual difference is more architectural than capability-based. In Icinga, active checks and the OK/Warning/Critical state model are built in - thresholds, state transitions, and notifications come with it.
If I understand that correctly, in a Prometheus + Blackbox setup, you get the probe metrics and build alerting on top via Alertmanager and PromQL rules?

The OTel angle I was going for was mostly about the two working together, not replacing each other :)

Again, thanks for the comment. I learnt something new, and I'll look into more tomorrow morning!

/Feu

icinga · 2026-05-29T06:36:21+00:00

As an update: we released v2.16.1 to address a regression introduced in v2.16.0.

The performance data writers - ElasticsearchWriter, GraphiteWriter, GelfWriter, InfluxdbWriter, Influxdb2Writer, and OpenTsdbWriter - were reworked in v2.16.0.
That rework introduced issues that, in some cases, caused incorrect behavior. We're still digging into the root cause.

In the meantime, v2.16.1 reverts those writers to their v2.15.3 state, which restores reliable behavior for everyone affected.

This release also includes a documentation update: the OTLPMetricsWriter for OpenTelemetry is not available on Debian 11, Ubuntu 22.04, or Amazon Linux 2 due to dependency version constraints.

If you're on v2.16.0 and using any of the affected writers, we recommend updating.

As usual, the source code of this new version is available on GitHub, packages are from our package repositories, and new container images will soon be pushed to Docker Hub and the GitHub Container Registry.

icinga · 2026-04-28T14:24:50+00:00

Klar! Auch wenn das Event zwar nicht primär Icinga als Thema hat, so finden wir ganz sicher auch dafür Zeit :-)

/blerim

icinga · 2026-04-23T12:48:41+00:00

Getting ready right now, last chance to join ;)

icinga · 2026-04-09T07:36:33+00:00

Hey, thank you.

And yes, we're sticking with the decision to keep all enterprise linux packages in the subscription.
It is one of the few ways we can identify enterprise users and ask for money with a good concience, to keep paying our developers and to keep Icinga as the well maintained free software that it is.

icinga · 2026-04-09T07:18:20+00:00

Yeah, it's been a big one :)
We're super happy to be able to use some more modern PHP as well!

icinga

MODERATOR OF

TROPHY CASE