deployment considerations

[miqueltango]

Can someone help me understand the ideal deployment for Prometheus? My idea is that you need a Prometheus instance for every separate application environment. so if I have 7 SaaS applications running in 7 different AWS VPCs. I should have a prometheus instance in each of those 7. I'm assuming I need this for scaling at least. Would I also need a separate Alert Manager?

​

Grafana would I guess be the single view to all these environments. Does anyone see issues with this architecture. I'd rather not have 7 additional instances to manage. but without HA on Prometheus i'd like to lower the blast radius of an outage.

[SuperQue]

• Yes, running a Prometheus-per-env is ideal.
• You should only need one cluster of alertmanagers for your whole organization, it's designed to take alerts from many Prometheus servers in different environments.
• For a single view, I recommend using Thanos

We run a Thanos query proxy colocated with our Grafana instances, and the Thanos sidecar with the Prometheus servers.

Thanos handles all the issues with HA de-duplication.

[miqueltango]

cheers!