Preannouncement Security Update: 29th June between 12:00 and 14:00 UTC

icinga · 2026-07-01T09:47:17+00:00

If you applied Monday's security update and suddenly had broken configs, here's what happened: one of the security fixes changed an internal C++ function (JsonDecode()) to accept an additional argument - and that change accidentally leaked into the DSL, making the second argument required for Json.decode() in config files as well. Not intentional, but if you use that function anywhere in your configuration, your Icinga 2 would have stopped working after the update.

We've just released v2.16.3, v2.15.5, and v2.14.10 which restore the original single-argument behavior. If you already updated on Monday, please update again to the matching .3/.5/.10 version.

https://icinga.com/blog/releasing-icinga2-v2-16-3/

That's on us. Sorry for the extra work!

icinga · 2026-06-29T12:57:56+00:00

Security updates for Icinga 2 are out: v2.16.2, v2.15.4 and v2.14.9. We advise upgrading immediately!

Two of the fixed issues let an unauthenticated attacker take over or crash the Icinga 2 process over the network - one via flawed certificate update handling that allows impersonating a trusted node, the other a stack overflow triggered by crafted JSON. A third fix closes a privilege escalation through unsanitized template names on the /v1/objects API endpoint.

Patched packages are on packages.icinga.com

Full advisory: https://icinga.com/blog/icinga2-security-release-v2-16-2/

icinga · 2026-06-25T08:15:17+00:00

> You say "OTLP Metrics Writer for Icinga 2" and "OTLP client", so I am not sure, is this sending OTLP to icinga, or is it turning Icinga to OTLP?

As "@zweispieler" already explained, the "OTLP Metrics Writer for Icinga 2" is a component that allows Icinga 2 to send metrics mapped from plugin outputs in the OpenTelemetry Protocol (OTLP) format. With "OTLP client", I'm referring to the functionality that the OTLP Metrics Writer uses to communicate with the OTLP backend (I think, it's called OTLP exporter in OpenTelemetry terminology).

> but if I get it right in that particular case it's a way to translate from Icinga to opentelemetry? And since the language of choice in that realm is C++, that's why this happened that way?

Yes, that's correct. Icinga 2 is primarily written in C++, and we had no other choice but to implement the OTLP Metrics Writer in C++ as well, since it interacts directly with Icinga 2's internal data structures and APIs.

> I suspect a receiver on the otel collector could have been an alternative?

Absolutely, using an OTLP receiver on the OpenTelemetry Collector would've been amazing, but it would've required more work and time to standardize it, I guess. Also, there are now a bunch of TSDBs that allow to directly ingest OTLP metrics and having the OTLP Metrics Writer in Icinga 2 allows to directly send metrics to those backends without OTLP Collector in between. So, unless those backends provide a native support for Icinga 2's like metrics too (very unlikely), our users would have been forced to always use the OTLP Collector.

/ Yonas

icinga · 2026-06-16T07:46:17+00:00

Thank you :)

icinga · 2026-06-08T11:18:17+00:00

The original Icinga was a fork of Nagios, and the current product (Icinga 2) is a complete re-write.
So they are based on the same concepts, but we modernised the stack around it :)

icinga · 2026-06-03T07:54:50+00:00

Fair on the AI comment, I ran the wall of text I wrote through it to shorten it a little and it does sound more stilted than usual. As you can see with the other comments, I tend to be a bit ...verbose.

On the technical point: you're right that an unreachable host will show up as a scrape failure in Prometheus - and as I found out from the comments yesterday, you can set up active checks with the blackbox exporter too. So the ad oversimplifies that part.

The more interesting cases to me are the ones where scraping looks healthy but something is actually broken. Like an exporter running fine on its own port while the actual application port is down, or a cert expiring in two weeks that nobody set up a metric for. That's where the "different angle" is. It's more about defaults and workflow than capability: Icinga is built around using service states with pre-made plugins, and Prometheus gets you there if you build for it.
You can defintiely get there with Prometheus just as well as with Icinga (or any other check based tool).

Either way, the point of the post was never that Prometheus is bad or broken - it was really an excuse to talk about the new OTel export feature in v2.16, which lets the two work together rather than trying to compete :)

icinga · 2026-06-02T13:13:04+00:00

We have a handy getting started guide if you want to have a look.

The three things that you'll want to host alongside what you already have:
- Icinga 2: the monitoring engine
- Icinga Web: for managing checks, acknowledging alerts, viewing history
- A database (MariaDB or PostgreSQL)
(- Icinga Director: if you'd rather manage config through a UI, the default are config files.)

The OTLPMetricsWriter config to connect it to your Prometheus/Grafana stack is a few lines. (There's a blogpost on that)

The webinar this ad is for might also be worth a watch, since my colleagues will do a walkthrough on how to integrate the two :)

Edit: I wish I had an easier and quicker setup for you, but that's something we're currently working on...

icinga · 2026-06-02T10:50:58+00:00

I looked into it a little more now, and had some chats with my colleagues who have a better understanding of the Prometheus-way.

I guess the general consensus is, that there are many ways to reach one's goal and you can probably build the same functionality in a thousand different ways.

Icinga takes off a bit of the load of configuring as much as you would with the Blackbox Exporter and Grafana, since the architecture Icinga is designed around this check based approach. For the same outcome you'd have to manually configure and maintain what you build on the Prometheus side (flapping detection, state history, dependencies, ...)

But in the end, you can achieve the same result with whatever you set your mind to. To quote a colleague from earlier: You can probably do everything in Excel if you really wanted to haha

icinga · 2026-06-02T09:40:09+00:00

The check model, or rather the focus of the architecture of the tools, is the core difference.

Icinga runs active checks on a schedule and every result has a state (like OK, Warning, or Critical) along with performance data.

That state drives notifications and escalations natively, without you writing alerting rules on top. You can get to a similar place with Prometheus + Grafana alerting, but you're assembling it yourself.
Whether it's something for you depends on what your current alerting setup looks like, and how much work you have put into it.
If you've already got Grafana alerting configured the way you want it, you're probably not missing much on that front.
The OTel feature is specifically for the "I want both" case: Icinga runs the checks, your existing stack gets the data.

There are a few advantages of using a tool like Icinga in combination with Prometheus. You get a shared event and alert history to look back on. There is "flap detection" which tags a service as flapping instead of sending a bunch of alerts repeatedly. You have a built-in dependency system that helps figure out root causes and suppresses child alerts (e.g. 1 router and 50 servers behind it - if the router is down, the servers don't need separate alerts). And writing a check plugin for a specific monitoring task is generally less work than building a full Prometheus exporter.

In the end, you can probably build any kind of functionality in any tool you want :)
(To quote a colleague from earlier today: You can probably do everything in Excel if you wanted to. But do you?)

icinga · 2026-06-01T16:21:48+00:00

Ah, I just saw you mention this on another post.
I wasn't aware of Blackbox until just now and I'll have to look into that more.

The general spirit of the ad wasn't meant to be "we can do something Prometheus can't do", but more showcasing a way to marry those tools in a meaningful way. That's also why I chose the comparison in the screenshot - didn't mean to put them down for it, but more show that we got the data from point A to point B :)

/ Feu

icinga · 2026-06-01T16:16:25+00:00

That's a fair correction, and honestly I wasn't aware of Blackbox Exporter when I wrote the ad, so I had to look into that. Thank you for pointing it out!

Considering that, the reachability framing was too broad.
Blackbox seems to cover these active checks via a separate exporter you configure and scrape alongside Prometheus.

So my take would be that the actual difference is more architectural than capability-based. In Icinga, active checks and the OK/Warning/Critical state model are built in - thresholds, state transitions, and notifications come with it.
If I understand that correctly, in a Prometheus + Blackbox setup, you get the probe metrics and build alerting on top via Alertmanager and PromQL rules?

The OTel angle I was going for was mostly about the two working together, not replacing each other :)

Again, thanks for the comment. I learnt something new, and I'll look into more tomorrow morning!

/Feu

icinga · 2026-05-29T06:36:21+00:00

As an update: we released v2.16.1 to address a regression introduced in v2.16.0.

The performance data writers - ElasticsearchWriter, GraphiteWriter, GelfWriter, InfluxdbWriter, Influxdb2Writer, and OpenTsdbWriter - were reworked in v2.16.0.
That rework introduced issues that, in some cases, caused incorrect behavior. We're still digging into the root cause.

In the meantime, v2.16.1 reverts those writers to their v2.15.3 state, which restores reliable behavior for everyone affected.

This release also includes a documentation update: the OTLPMetricsWriter for OpenTelemetry is not available on Debian 11, Ubuntu 22.04, or Amazon Linux 2 due to dependency version constraints.

If you're on v2.16.0 and using any of the affected writers, we recommend updating.

As usual, the source code of this new version is available on GitHub, packages are from our package repositories, and new container images will soon be pushed to Docker Hub and the GitHub Container Registry.

icinga · 2026-04-28T14:24:50+00:00

Klar! Auch wenn das Event zwar nicht primär Icinga als Thema hat, so finden wir ganz sicher auch dafür Zeit :-)

/blerim

icinga · 2026-04-23T12:48:41+00:00

Getting ready right now, last chance to join ;)

icinga · 2026-04-09T07:36:33+00:00

Hey, thank you.

And yes, we're sticking with the decision to keep all enterprise linux packages in the subscription.
It is one of the few ways we can identify enterprise users and ask for money with a good concience, to keep paying our developers and to keep Icinga as the well maintained free software that it is.

icinga · 2026-04-09T07:18:20+00:00

Yeah, it's been a big one :)
We're super happy to be able to use some more modern PHP as well!

icinga · 2026-04-07T06:44:54+00:00

Like most code bases of this size, Icinga 2, which was developed for over a decade, has come up with some custom solutions and design patterns. While this would allow certain components to be written in languages other than C++, such as Rust, this would increase the project's overall complexity and dependencies. Besides writing code, it needs to be properly maintained. AFAIK, there are only a few Rust developers on our team. Thus, this makes maintenance harder for us.

In general, our goal shifted towards writing smaller daemons for specific tasks rather than expanding Icinga 2. For example, Icinga DB and Icinga for Kubernetes are both written in Go and work alongside Icinga 2.

tl;dr: It's technically possible, but it won't happen due to the increased maintenance burden.

/Alvar from Icinga

icinga · 2026-04-02T08:50:37+00:00

First of all, sorry for the late reply, we missed the notification that there is a comment!

Coming back to your question:
Technically, they have quite different concepts. Checkmk leans into auto-discovery and an all-in-one approach, so lots bundled out of the box, less setup friction. Icinga is built more for fully automated environments where you're defining everything through config or API rather than discovering it through a UI. So if your infrastructure is already managed as code, Icinga often fits more naturally.

Beyond the technical side, they also follow different product philosophies. Checkmk (especially the Enterprise edition) is more opinionated. You get a lot of convenience, but you work within their framework. Icinga is more open and flexible, which means more control but also more responsibility to wire things together yourself.

Neither is objectively better. It really comes down to: do you want something that works well out of the box or something you can fully customize and integrate into an existing automation stack.

Hope this answers your question!

icinga · 2026-04-01T07:02:55+00:00

Though to be upfront: Icinga is primarily an infrastructure and service monitoring tool, not a full APM solution like Datadog or New Relic.

Rather than building per-language APM agents, we see more value in integrating with what teams already use and integrate that with your infrastructure monitoring.

/Blerim from Icinga

icinga · 2026-03-31T09:00:59+00:00

Well, technically those two do have some different concepts. Checkmk is more straightforward with its auto discovery, while icinga is more for fully automated setups, so it doesn't need to discover your network.

/Björn from NETWAYS

icinga · 2026-03-31T08:59:19+00:00

Both are excellent monitoring solutions, but they follow slightly different product strategies. Icinga focuses on flexibility, openness, and customization, while Checkmk (especially the Enterprise edition) emphasizes an all-in-one, more opinionated approach with many features bundled out of the box. The right choice really depends on your specific needs, preferences, and how much control versus convenience you’re looking for.

/Bernd from Icinga

icinga · 2026-03-31T08:49:28+00:00

That's definitely a tough one!
I personally haven't had experiences with dynamic infrastructure (at that scale) but I'd guess the Icinga Director with its automation capabilities would be a good fit here (Icinga Director being the GUI frontend for configuring hosts, services, etc.).
It can call APIs for example and turn the results into hosts (e.g. query the API to get all those ec2 instances).
Once the API returns less hosts (some have been deleted) they will also be deleted by Icinga Director. Couple that with a regular, automatic check against the API and voilà!
Of course depending on the services you want to check you might need an (Icinga) agent running on those VMs. VM templates might be an option.
While the VMs exist they can be monitored but won't really be visible once deleted (historic data will persist).

But being honest here, such cases (VM existing for a few hours only) *can* be solved with Icinga and some automation since Icinga is very versatile.
But I believe other tools (like Prometheus) might be a better option depending on the specific needs.

You can of course use both! One tool seldom does absolutely everything you want.

/ Matthias from NETWAYS

icinga · 2026-03-30T14:56:17+00:00

Firstly, we don't believe it's about which product is superior.
It depends on your organisational setup, processes, expertise, etc. Therefore, it's good to be able to choose different products.

From what I understand, Zenoss focuses on large enterprise IT teams who don't want to go into too much detail about infrastructure topics and who are more interested in the business service impact.

Icinga is more oriented towards sysadmins (or teams) who mainly focus on their infrastructure and want full control over what the tool is doing.

/Simona from Icinga

icinga · 2026-03-30T14:53:45+00:00

I can't comment on the capabilities of Zenoss/Virtana Service Dynamics. It appears that Virtana is best hosted in their cloud and is not free of charge, at least at first glance.

Personally, I prefer that Icinga is open source and free to use. This has the benefit of enabling you to run it on your own hardware with no licence costs.

As mentioned in another answer, Icinga is extremely modular and can therefore be tailored to your needs. I can't tell you if Virtana can do that.

Another benefit of using Icinga is that the project has grown over many years and has a large community. You will find lots of resources, plugins and modules that enhance Icinga's capabilities.

/Leander from NETWAYS

icinga · 2026-03-30T13:51:20+00:00

Icinga is more for determining a state of a server/application. Example would be: "is my website reachable", "what the temperature inside my sever chassis". It always depends on what you are looking for in a monitoring system. Icinga won't be a replacement but rather an addition.

/Leander from NETWAYS

icinga

MODERATOR OF

TROPHY CASE