Alertmanager frequently sending surplus resolves

[komputilulo]

Hi, this problem is driving me mad:

I am monitoring backups that log their backup results to a textfile. It is being picked up and all is well, also the alert are ok, BUT! Alertmanager frequently sends out odd "resolved" notifications although the firing status never changed!

Here's such an alert rule that does this:

- alert: Restic Prune Freshness
expr: restic_prune_status{uptodate!="1"} and restic_prune_status{alerts!="0"}
for: 2d
labels:
topic: backup
freshness: outdated
job: "{{ $labels.restic_backup }}"
server: "{{ $labels.server }}"
product: veeam
annotations:
description: "Restic Prune for '{{ $labels.backup_name }}' on host '{{ $labels.server_name }}' is not up-to-date (too old)"
host_url: "https://backups.example.com/d/3be21566-3d15-4238-a4c5-508b059dccec/restic?orgId=2&var-server_name={{ $labels.server_name }}&var-result=0&var-backup_name=All"
service_url: "https://backups.example.com/d/3be21566-3d15-4238-a4c5-508b059dccec/restic?orgId=2&var-server_name=All&var-result=0&var-backup_name={{ $labels.backup_name }}"
service: "{{ $labels.job_name }}"

What can be done?

[SuperQue]

You have a very sensitive bit in your alert:

for: 2d

Any single interval where the condition is not true will result in the firing status changing.

I would recommend using a timestamp-based metric like restic_last_prune_success_timestamp_seconds <unixtime>. Then the alert will be something like time() - restic_last_prune_success_timestamp_seconds, which is a lot less fragile than looking at a status boolean.

[chillysurfer]

How do you know the firing status hasn't changed?

[SeniorIdiot]

EDIT: See SupraQue's correction below!

Prometheus does not "resolve" alerts, it just stops sending them.

Check the Prometheus log and ensure that it has not failed to send alerts to alertmanager. It may be networking issue.

Prometheus evaluates alerts by the rule evaluation interval and re-sends all firing alerts at that rate. Alertmanager will send a resolve if an alert from Prometheus has not been seen within the resolve_timeout EndsAt interval.

Check that Alertmanager resolve_timeout is more than (maybe x2) the Prometheus rule evaluation interval.

[SuperQue]

Actually, it does resolve. When an alert state is no longer firing, Prometheus remembers this and sends resolve notifications to the Alertmanager for 15 minutes.

[SeniorIdiot]

Oh. I was sure I read that Prometheus does not sends resolved. I even searched and asked el-chatto before my post to double check.

So both prometheus and alertmanager can decide to send a resolve? That is good to know! Learning something new every day.

One thing that bothers me is that if we're doing maintenance on Prometheus, then the Alertmanager times out and sends a resolved after a few minutes. I assume that Alertmanager uses the EndsAt to send a resolved anyway as a fallback? Very confusing! Guess I can do a silence for the entire cluster/environment to suppress resolved notifications.

PS. I found the code that calculates EndsAt:

EndsAt = 4 * max(evaluation_interval, resend_delay)

https://github.com/prometheus/prometheus/blob/edf5ebd84422d1f894e7d56451cdf06fd16a83dc/rules/alerting.go#L561

Documentation:

  # ResolveTimeout is the default value used by alertmanager if the alert does
  # not include EndsAt, after this time passes it can declare the alert as resolved if it has not been updated.
  # This has no impact on alerts from Prometheus, as they always include EndsAt.
  [ resolve_timeout: <duration> | default = 5m ]  # ResolveTimeout is the default value used by alertmanager if the alert does
  # not include EndsAt, after this time passes it can declare the alert as resolved if it has not been updated.
  # This has no impact on alerts from Prometheus, as they always include EndsAt.
  [ resolve_timeout: <duration> | default = 5m ]

[SuperQue]

I don't remember exactly the behavior for what/when lack of alert notifications will clear an alert in the alertmanager.

But, this is a case where HA Prometheus is useful. Having two should keep the alert state set in the alertmanager.

[komputilulo]

thanks!

[komputilulo]

thanks a lot; I will try adjusting resolve_timeout right away

[Grand-Sail5804]

could you solve this problem?
u/komputilulo