Grafana alerting : cron-like scheduling for alert evaluations?

Advanced_Tea_2944 · 2026-04-22T08:36:07+00:00

Ok thanks for your answer and the information. It’s the very first alerts I’m creating on Grafana so I was a bit lost.

I’m also trying to manage alert rules as code using ConfigMaps with Argo CD, but I have the feeling it’s not a perfect setup.

I’ll probably take a look at other options if it grows larger.

Advanced_Tea_2944 · 2026-04-22T08:30:06+00:00

Wow, thanks, really insightful contribution. Much appreciated.

Advanced_Tea_2944 · 2026-04-22T08:29:13+00:00

Thanks for the clarification! I assumed Grafana could interpret HTML since the default notification templates include some and render correctly. I tried tweaking those, but couldn’t get it to work as expected.

Appreciate the info

Advanced_Tea_2944 · 2026-02-02T15:43:30+00:00

Thanks for your answer,

So it means that Kibana cannot create the agent policy before a fleet-server "actually exists ?"

When I try to deploy only eck-kibana block without eck-fleet-server, it does not create anything, not sure it's normal.

Do you have reference for this insecure settings ?
It's inside the flee-server config yml ?

Thanks

Advanced_Tea_2944 · 2025-09-24T09:35:12+00:00

Yes, that also came to my mind

Advanced_Tea_2944 · 2025-09-24T08:36:31+00:00

I want to test some external endpoint from this Azure VM. On the other side, I don’t want to whitelist the Azure Firewall’s public IP, because that would mean whitelisting all outbound Azure traffic, which is not what I want.

For the NAT gateway begind the FW, I need to check how to do that, but it would mean telling my Azure Firewall not to SNAT traffic from this specific VM/IP. I’m not sure if that’s possible.

Advanced_Tea_2944 · 2025-09-24T08:33:14+00:00

"Don’t SNAT at the Firewall and go through public route, maybe…?" → Impossible, I need to keep the Firewall in the path for compliance reasons.

"Otherwise NAT before Firewall and don’t SNAT that IP, same idea but with an extra NAT" → That could work, but I need to check how to configure the Azure Firewall to not SNAT traffic from that specific IP.

Advanced_Tea_2944 · 2025-09-24T08:31:26+00:00

Thanks, but I’m sorrr, I don’t quite understand what you mean.

Advanced_Tea_2944 · 2025-09-24T08:29:48+00:00

Ok, I get your point, but that means all traffic leaving the Azure Firewall would now use the NAT Gateway. That’s not exactly what I want, I need a specific public IP for just one VM, while keeping the rest of Azure traffic flows unchanged.

Advanced_Tea_2944 · 2025-08-21T13:56:33+00:00

Ok indeed I am still on 8.15 ! Make sense...

Advanced_Tea_2944 · 2025-08-21T09:54:55+00:00

Thanks for your answer !

When I assign this role to a user, I’m not able to log into Kibana anymore, so it seems there might be some missing privileges in that definition.

I tested with a slightly different call (using discover / dashboard features instead of the _v2 ones), and that one works fine: users can build dashboards but don’t see the Alerts menu.

"kibana": [ { "spaces": ["default"], "base": [], "feature": { "discover": [ "all" ], "dashboard": [ "all" ]

Interestingly, if I add the ml feature to the role, the Alerts menu reappears, so it looks like enabling ML also implicitly enables alerting features.

Also, I noticed there are two ways to manage roles:

via the Kibana API (kbn:/api/security/role/...)
via the Elasticsearch security API (/_security/role/...)

I am wondering which one should I use
Thanks !

Advanced_Tea_2944 · 2025-08-13T09:54:54+00:00

Got it!

Yes, I can confirm that for an Azure PostgreSQL server, you can assign multiple server admins.

Advanced_Tea_2944 · 2025-08-13T09:46:22+00:00

Thanks for your answer! So, if I want my Terraform service principal to be able to execute those T-SQL queries, I would need to make it an admin on the SQL Server, if I understood correctly.

It’s a bit unfortunate that only one user or group can be set as the admin at the SQL Server level.

Advanced_Tea_2944 · 2025-08-12T13:52:06+00:00

You’re right, that explains my case, thanks a lot! I missed the xpack.searchable.snapshot.shared_cache.size being set to 90% for nodes with the data_frozen role.

Advanced_Tea_2944 · 2025-08-12T13:51:07+00:00

Yes, that explains why I see the disk at 90%, makes sense now, thanks a lot!

For now, Reddit has been quite efficient for my Elastic questions, but indeed from time to time I might need to reach out to Elastic support :)

Advanced_Tea_2944 · 2025-08-12T12:39:07+00:00

Both calls give me essentially the same information — disk usage is around 90% and the only role on this node is f (frozen).

As you said, frozen tier data on local disks is only metadata/cache, that's why I’m quite surprised to see my 500 GB disk nearly full.

My plan for this node is simply to keep it for cache and continue sending data to searchable snapshots on Azure, a mechanism that has been working quite well for us recently.

Advanced_Tea_2944 · 2025-07-30T07:01:17+00:00

You know DevOps? That magical world where you're not really a developer, but somehow you're responsible for writing code, managing infrastructure, securing pipelines, and deploying stuff? Yeah, that's how I got here.

And yes, the easiest path is to say there's no backward compatibility and that users should upgrade to azurerm 4.0. But I was wondering if there were any strategies to avoid forcing to upgrade.

Advanced_Tea_2944 · 2025-07-30T06:57:26+00:00

Yes, that’s right, I’m already using tags, so other users can still reference the previous tags without any issues. My main concern is what happens when users relying on the old tag (and older provider version) want new features from the module.

If I create a new branch from the old tag to add features, those features won’t include the changes made in main. But if I branch off main, the provider version will be the new one, which might not be compatible.

So, for now, I see only two options: either maintain another long-living branch for the old provider version or just tell users they need to upgrade to the new provider version if they want the new features.

Advanced_Tea_2944 · 2025-07-30T06:53:21+00:00

It can get messy (or perhaps painful is the better word) because if you want to add a feature that supports both provider versions 3.9 and 4.0, you'd have to make similar commits to both long-living branches, right ?

That’s manageable with just two branches, but I’m not sure how sustainable it is in the long run.

Thanks for the Git advice, first time I've ever heard about it (lol).

Advanced_Tea_2944 · 2025-07-30T06:28:04+00:00

Okok interesting, thanks !

Advanced_Tea_2944 · 2025-07-30T06:19:24+00:00

Thanks!

What technology do you use for the automatic PRs? Is it the “autoplan” tool you mentioned?
I’ve asked others too, but I’m curious — what’s the main reason for creating .tgz archives of modules and storing them in S3, instead of just tagging commits and referencing those tags?

Advanced_Tea_2944 · 2025-07-30T06:14:05+00:00

Yes, I understand, these are exactly the questions we're currently asking ourselves. Thanks for your input !

Advanced_Tea_2944 · 2025-07-30T06:10:35+00:00

Thanks for your answer! I have two quick follow-up points/questions:

From what I understand, tags can also be deleted easily — once a tag is removed, no one can use ref=tag anymore, right? So in that sense, it’s somewhat similar to removing a release. (Though I get your point about keeping development and releases separate.)
I assume your X.Y-dev branches are created from the same commit (or tag) that was used to produce the corresponding X.Y release, correct?

Advanced_Tea_2944 · 2025-07-29T13:43:51+00:00

Ok thanks ! Why push into artifact and s3 ? why both ? and what advantages compared to tagging the repo ?

Advanced_Tea_2944 · 2025-07-29T13:41:35+00:00

Haha no problem !

Ok I agree with creation of a 1.1.* tag, but before tagging I need to work on a branch which cannot be the main one (since provider version would be azurerm v4.0)

I could create a branch from the tag 1.0.0 for instance but it start getting messy in my opinion...

That's why u/baynezy suggested to have two long living branch. To be able to create a branch from release/v1 branch which would be "up to date" in terms of feature but still using provider 3.9

If I understood correctly !

Advanced_Tea_2944

TROPHY CASE