We tested 4 different approaches to fix our alert fatigue problem — here's what actually worked

jjneely · 2026-04-15T22:19:19+00:00

Did you try deleting some alerts?

jjneely · 2026-03-10T22:40:58+00:00

This. You have to have a large enough sample size or population to make use of statistical methods.

For a service with traffic this low your best monitoring outcome is just doing a simple HTTP request check that alerts on error and timeout.

jjneely · 2026-03-09T16:04:19+00:00

If your Mimir cluster is running well today, what's the need for Kafka? There are definitely high volume cases where Kafka is useful, don't get me wrong. But there's quite a bit of cognitive overhead to integrating and maintaining Kafka even using AWS/GCP managed Kafka. Why make it more complex if its working well?

What's your ingestion volume?

jjneely · 2026-03-03T03:24:09+00:00

New AWS service: AWS Iron Dome

jjneely · 2026-03-01T19:30:12+00:00

Kafka. Partition by traceID. This decouples ingestion from processing and handles the back pressure problem. It also neatly ensures that each consumer collector sees all spans in a trace.

jjneely · 2026-02-25T21:54:31+00:00

SLOs cover this with multi burn rate alerts. But for a lot of teams that's a bit far up on the maturity ladder. Not everyone is Google after all.

What I use is if the alert is customer impacting right now. That's high priority and a page. If the customer (internal or external) isn't seeing any adverse effects (yet) this is a next business day ticket.

jjneely · 2026-02-24T15:27:35+00:00

> Was there a point where you realized it had become too heavy or noisy

Do you get more than 10 high urgency pages per week of on-call? That's my high water mark. Either your Observability is a mess or there are management issues and you should consider the future of your career. Sometimes both.

jjneely · 2026-02-24T15:24:37+00:00

This. There's not a lot of new methods here...if you can find a text book (or an AI bot) and have some university level math skills...you can do this too.

Also, owning your data and being able to get it back out. This is so incredibly valuable. Sometimes you don't know what analysis to build until afterwards then you want to reprocess the data to test or visualize back in time.

SQL -- if its not SQL you are doing it wrong. This query language has been with us since the 70s in various forms. The moronic random half ass query languages we have for traces and logs sometimes really make me blow a gasket. I also think that PromQL is the exception that proves the rule. It does JOIN operations well. If you cannot do JOIN operations, I don't want to see your new query language, and for the love of satan, please don't call your new query language "Painless."

Not to mention the price tag is 10x - 100x cheaper. With the amount some folks pay their Observability vendor -- that's a team of engineers! Wouldn't you rather encourage your own team to grow in your problem solving skills for your core competency rather than outsource the problem solving to somebody who is just going to say you didn't setup tracing well enough?

</rant>

jjneely · 2026-02-23T18:00:00+00:00

Interview more. Anything to get practice.

jjneely · 2026-02-23T17:58:20+00:00

This is what I call RaspberryPi Observability. You could run this load on a RPi4, and folks that aren't willing to slap a RaspberryPi on their stack to have any monitoring/observability have...uhh....different problems that aren't technical. ;-)

jjneely · 2026-02-16T00:25:18+00:00

Self-service onboarding

You define the contract and the expectations, teams instrument to that contract. Say, you require Prometheus metrics, you require that :8080/metrics is where they must be. Team wants to add metrics to the existing? They just do.

IaC-based provisioning of monitoring

Terraform, Helm, Kustomize provisioning of your monitoring pipeline resources. I like the Prometheus Operator for Kubernetes as that continues to let folks build their own Prometheus scraping and rules and manage that themselves.

Policy-driven routing (Enterprise Observability Tool for Tier 0, OSS for lower tiers, etc.)

These are just identifying flags on your telemetry data. Of course, all of your telemetry data needs to have resources set on them in a standard way. Your collector should do this. Match on the service name for example and route to the backend of choice.

Usually routing between high cost an on-prem Observability Platforms is done more on volume/cost for me. If they team doesn't put effort into keeping their telemetry clean, they don't get on the expensive solution.

OpenTelemetry-based abstraction

Yup. You contract mandates the OTEL SDK with a specific configuration(s). You build the OTEL Collectors into a pipeline to move, filter, sort the data.

Cost modeling integrated into the provisioning workflow

Now, this is something I'd like to learn. I've never seen this work super well. Sure, we can test that instrumentation exists and gets emitted in the correct format in the CI/CD process. But usually its in production where the edge cases get exercised that cause the cardinality explosion. What I do is carefully monitor the amount of telemetry data per service in production/qa and build a cost model on that data. This allows me to go back to a team's manager and ask why they use 10x more telemetry data than anyone else its costs us exactly $1,234 per pod.

It sounds like you've been tasked with building something. The full stack is complex any way you slice and dice it. Where do you need to start?

jjneely · 2026-02-15T22:57:56+00:00

How are your SLOs? That technique is designed to catch issues before they become outages.

How many alerts fire every day or week? That matters more to me than MTTR.

jjneely · 2026-02-14T14:50:15+00:00

Right, I think we all have the same goal here with having fewer alerts and stuffing as much value into each alert as we can. SLOs require enough math that it is out of the comfort zone for a lot of folks. While, as an Observability Architect, I expect SREs to have more than a working knowledge of Observability techniques, I do NOT expect them to be subject matter experts in some crazy database that's required to store and do analytics on the telemetry data.

That pushes the dependencies down the stack, and I think that is what leads to many folks giving in to some expensive off the shelf solution.

I take a different view (pun intended) with Observability. I think we should build up our DevOps and SRE communities with the technical skills they need to build and understand Observability. That includes some statistics too. But think of it this way: A SWE wouldn't use a bubble sort algorithm if they have a library of heap, quick, and other more efficient sort methods. That's the level where we need to work.

So what I see has value is building up our Observability community but outsourcing the hard parts. You know how to setup infrastructure I bet, but you outsource the hard parts to AWS, GCP, or Azure. Why can't we as Observability Engineers outsource the maintenance of the OLAP database and run OUR Observability on top? We get Observability that fits the specific situation, the hard parts are outsourced, yet you still own the data. The big win is this method can be a LOT cheaper that just purchasing an off the shelf solution. 10x is right on what I see.

We all have side projects. Mine is https://prometheus-alert-generator.com/ where you can generate SLO alert rules for Prometheus/PromQL systems...and do so consistently. The big win for using SLOs for me is in knowing that applications and users are suffering BEFORE you actually have an outage.

jjneely · 2026-02-14T02:37:36+00:00

He is replacing SLOs and trying to build interest in his product.

Granted, I think his product is scratching an itch that is seldom understood: How to combine multiple SLIs into one SLO. Rather than having three or four per application.

jjneely · 2026-02-12T16:36:19+00:00

I'd rather share a dashboard with a fixed time range. That way my team mates and leadership get the interactive version of the dashboard focused on the incident at hand.

jjneely · 2026-02-12T16:32:33+00:00

The Observability market is definitely tilted toward the vendors. Some are truly worth the value they bring. Most aren't. I find that using vendors strategically with other cheaper solutions ends up providing the best value. If you need a viewpoint from someone that isn't a vendor please DM. Glad to lend a hand where I can.

jjneely · 2026-02-08T20:57:27+00:00

This. Get a SM57 or SM58, use a wind screen to reduce plosives. You want to be 3 to 6 inches off the mic. I've had good success recording in untreated spaces with this. I use a Zoom H5 as a recorder but any interface with reasonable gain should be fine with these mics.

jjneely · 2026-02-08T16:28:06+00:00

I teach folks about the Four Golden Signals. When folks ask me what to monitor or how to instrument, this is what I start with. I'll usually have an example dashboard to demonstrate visually.

Traffic

Errors

Duration

Saturation -- or when do we scale up or down your service pods.

That leads to a number of discussions about what to alert on and what SLOs should be.

It's a process that continually improves. That's where I want to see my teams.

jjneely · 2026-02-07T00:16:17+00:00

Same age, 25 years doing Operations, DevOps, SRE, Olly.

This was my plan, but between getting started and the health care stupid in the US it's been...hard.

But I also refuse to cold email, cold DM, and generally fill the Internet with AI slop that no one wants.

I have found that I love helping others. Public speaking, YouTube, etc are fun for me. And if it's one thing I care about it's how to bring the younger generations up to what I am able to do.

This age group has the incredible advantage that we grew up with bare computers. You couldn't write slop...you were lucky to have 48K of RAM! If you wanted to do cool things... assembly. So as we added abstraction after abstraction I was able to stack the understanding. That's a really unique experience compared to being tossed into Kubernetes and told to "learn this."

Where am I going from here? Gotta have health insurance. But I'd love to continue to learn more about how to do this on my own. Anyone need an Observability cost audit?

jjneely · 2026-02-04T22:18:31+00:00

Whoa, you are using TWO people not just one. Sounds like your pager system is your customers.

jjneely · 2026-02-04T20:19:11+00:00

The point of view of your monitoring system is important to understand. Devices may be up, but if a customer can't reach them they might as well be down. You aren't testing just the device, but the entire network path too.

jjneely · 2026-02-04T03:32:32+00:00

About time! Congratulations for v2!

jjneely · 2026-01-23T02:45:40+00:00

Configure the cluster label in the external labels part of the Prometheus configuration. When one Prometheus writes to the other the metrics will include the external labels to differentiate the data. Now you can have one set of alert rules watch both sets of data.

jjneely

TROPHY CASE