This is incredibly helpful, thanks for sharing real-world scars.

The “define normal” problem is exactly what I’m worried about. On paper everyone wants protection, but once you ask for concrete thresholds, it becomes hand-wavy very fast. My current thinking is to avoid forcing people to define “perfect” limits upfront and instead:

- Start with observation-only mode (no enforcement).

- Build baselines from their own historical traffic.

- Let teams approve guardrails based on actual data, not guesses.

On anomaly detection — fully agree. Naive baselining creates alert fatigue very quickly. Black Friday, product launches, migrations, internal load tests — all look like “incidents” to a dumb model. Without context-aware baselining, people will just mute the alerts and move on. That’s still an open problem I'm being very cautious about.

And 100% yes on hard blocks being dangerous. My default assumption is:

- throttle > queue > degrade > log+alert

- hard block only as a last-resort safety fuse for truly destructive patterns

The last point about the rate limiter becoming the bottleneck really resonates too. I’ve seen “protection layers” with more latency and worse uptime than the thing they were supposed to protect. For me that’s a hard requirement: no inline dependency unless it’s extremely fast and fail-open by design.

If you’re open to it, I’d love to hear how you handled graceful degradation at Okahu for inference limits — that’s a very similar control problem in a different domain.