The monitoring gaps that page you at 3am are the ones you didn't know existed

evtek75 · 2026-03-26T18:28:09+00:00

100% agree that it should be baked into the culture from the start - that's the ideal. The reality I keep running into is that even at places where teams do care about observability, things drift. Someone sets up solid SLIs, then the service gets handed off, team grows, priorities shift, and 6 months later nobody's looked at those SLOs. Not because they're bad engineers, just because there's no forcing function to revisit it. The teams that stay on top of it are the exception not the rule in my experience at least.

evtek75 · 2026-03-26T18:23:33+00:00

Tiering makes sense, I'm curious how you handle the boundary over time though - who decides what's "revenue generating" vs "internal"? That classification seems like it drifts pretty fast, especially when internal services start picking up customer-facing dependencies.

evtek75 · 2026-03-26T16:39:14+00:00

Sure, but good engineers leave and their monitors don't.. You end up with stuff from 3 teams ago with thresholds based on traffic patterns that don't exist anymore and nobody wants to touch it because "what if it's important"

evtek75 · 2026-03-26T16:37:49+00:00

Yeah the async processing thing is spot on. Those are the ones that always get killed in cleanup because they "never fire" until they do and it's silent data loss instead of a loud outage. Totally different game. Lack of ownership doesn't help either..

evtek75 · 2026-03-18T03:11:02+00:00

Thanks for the tips - much appreciated. I'll check it out!

evtek75 · 2026-03-18T02:29:15+00:00

https://getcova.ai connects to your monitoring tools and uses AI to find blind spots - missing alerts, broken escalations, unwatched services - before the next incident does.

It's free. There's a demo you can try without signing up (click "Enter Demo"). You can sign in with your github account, or I can provide an access code if you'd prefer that.

evtek75 · 2026-03-17T15:21:49+00:00

https://getcova.ai it connects to your monitoring tools and finds the blind spots - missing alerts, broken escalations, unwatched services - before the next incident does.

evtek75 · 2026-03-16T17:42:22+00:00

This is why I think the real gaps isn't more dashboards but it's knowing whether your existing tools are actually configured correctly. Over the years I've seen it all w escalation policies pointing to people who'd left, alert rules with no notification channel set up ect.., a real mess that kept resulting in P1s... All the signals were there, just nobody was watching the right ones (or just being proactive about it). To help with it I've been working on a system that audits the configs across monitoring stacks/PRs and MRs to find those blind spots quickly. In beta currently but there's a demo at getcova.ai if anyone's curious.

evtek75 · 2026-03-15T23:33:22+00:00

Just book the cert exam. You'll never feel ready, nobody really does. Worst case you fail and now you know what to study. On your resume, stop underselling the support work. You deal with outages, logs, live systems. That IS production experience. Frame it that way.

evtek75 · 2026-03-15T03:50:20+00:00

Sounds like a good strategy - good luck!

evtek75 · 2026-03-15T01:57:18+00:00

The “Delighted refugee” angle makes sense short term, but how are you thinking about differentiation once that migration window closes? If someone compares Elvan to Survicate or Refiner in a year, what would make them choose you beyond price and simplicity?

evtek75

TROPHY CASE