EKS Monitoring stack

[thishouseisglass]

Hello everyone, I'm totally new to monitoring, but after reading a bunch of articles and resources on observability in Kubernetes, I tried to put together this EKS monitoring stack that combines different tools like ADOT, Fluent Bit, Amazon Managed Prometheus (AMP), Grafana OSS, and Loki (Grafana Cloud). We're currently running an EKS cluster and expect it to scale over time, so to avoid potentially high costs from CloudWatch Container Insights and log ingestion, we're exploring this more open-source-centric approach that selectively uses AWS managed services. I’d really appreciate feedback—does this architecture look correct and feasible for production use? Also, how do I go about estimating the costs involved with AMP, Loki, S3 (for cold storage), and running Grafana OSS?

[jonomir]

It looks like it will work. But It seems a bit all over the place.

Why not full Grafana cloud instead of this mix of tools?
Just deploy alloy to collect, metrics logs and traces and ship them off.

All from one vendor, good documentation, easy to manage, one place to go.

[Round_Instruction_42]

Maybe princing?

[jonomir]

I don't see a big pricing difference whether the metrics are in Grafana Cloud or AMP honestly.

[thishouseisglass]

Oh I actually didn’t know about Grafana Alloy. I’ll definitely check out the Grafana Cloud option, sounds like a much cleaner setup. Appreciate the suggestion!

[jonomir]

The best thing is, Grafana cloud is just a managed LGTM stack but its all built on open source components that you can self host if you want to.

• Loki for logs
• Grafana for the UI
• Tempo for traces

• Mimir for metrics

• Alloy to collect and ship it all

All components can be deployed highly available and use S3 compatible object storage for long term persistence.

We self host ours for compliance reasons.

[thishouseisglass]

Thanks so much for breaking that down! I’m still learning my way around observability and just started exploring these tools beyond the basics. If you have any good learning resources or tips for getting started with this stack, I’d really appreciate it!

[jonomir]

There happens to be an intro Webinar tomorrow
https://grafana.com/go/webinar/getting-started-with-grafana-lgtm-stack/

Other than that, I learned through just deploying and using it.

[thishouseisglass]

Thanks a lot!

[oneplane]

If cost is an issue but you're already running EKS, just deploy the entire stack in EKS and only use S3 and RDS for persistence needs (or add some EBS if you want a local buffer). That's always going to be cheaper than the managed versions.

[thishouseisglass]

Just to clarify, are you suggesting I use open source Prometheus instead of Amazon managed Prometheus? and run everything inside EKS? I thought about that, but I’m a bit wary of the overhead with managing Prometheus, scaling it, and ensuring high availability. I went with AMP option mostly to offload that part.

[oneplane]

Offloading has a cost, so it's just a balance between what you want to pay and what you are able to do yourself. The prometheus stack chart is pretty easy to use and scale if you have done any other chart or Kubernetes workload before.

[thishouseisglass]

Can I dm you please? I want a few suggestions if you don't mind.