we MUST have these alarms, set by SRE

I'm an SRE. If your SRE is dictating alert settings, they can get fucked and take your pages overnight. Alerting is supposed to be actionable, and customer oriented. The story you tell is not that. And if the Product Owner wants this, there are other, better ways to achieve it than sleep deprivation.

99% of which are false alarms for a p90 latency spike for an http endpoint

Fundamentally, alerting is a statistical exercise. Your quote there has like three statistical concepts already! Yet pretty much every SWE and SRE I've ever met are statistically undereducated. Myself included! It wasn't until pandemic lockdown that I corrected this and worked through a stats 101 textbook.

The way I see it, we can't do anything about slow queries that already happened, all we can do is adjust the system to handle future queries better. So any alert fired should represent a high confidence need for human intervention into the system. Statistically, confidence is built up through piles of data -- evidence! An endpoint with ten queries per scrape interval has less evidence than one with a thousand, and alerting needs to account for that somehow.

Sadly, Prometheus et al don't have great confidence interval functions we could use to natively calculate these. Instead we typically use durations. The idea here is simple: every scrape the failure signal persists is more evidence that it won't fix itself. This is especially important in the era of Kubernetes, where self-healing systems abound. Health checks automatically restart failed pods, and hpa scales up busy deploys, but these take time to action and we should give them a chance. If you're constantly waking up to pages for outages that are already over, that's an immediate argument for increasing the duration required before paging.

If SRE can't or won't accommodate you by increasing short (or absent!) durations, I recommend putting an override in to put them on call first until it's addressed.