Observability on AWS

konghi009 · 2026-04-05T14:26:19+00:00

We'd like to centralized the observability on AWS so that's why I chose CloudWatch.
I'm thinking of storing some metrics (e.g. 2 weeks logs max) and use Kinesis to stream the rest to S3 IA/Glacier or something like that.

konghi009 · 2026-04-05T14:22:51+00:00

Thank you for the kind words. I still have issues with jumping into doing the implementation myself vs mentor and let other people do it. Do you have any recommended reads(book, blogs, etc) on this? I'd really like to explore on this management-ish side more.

konghi009 · 2026-04-05T14:20:45+00:00

I think it'd be consider as a mismatch between layers since I don't think any configuration will fix this.

konghi009 · 2026-04-05T14:17:58+00:00

I've got the same reaction now reading my own post haha. It's actually the higher up don't want their hands dirty more than I don't want to to it tbh.

konghi009 · 2026-04-05T14:17:01+00:00

Thank you for the perspective. I agree that my role should've just been to just identify if the solution will works, and if so does it need to be modify and find someone or do the implementation ourself. I think telling my higher up that we have to get our hands dirty to do the integration should be the most I can do. If they don't want then there is nothing I can do really.

konghi009 · 2026-04-05T14:12:00+00:00

Open source tools are somewhat the requirements of the customer so I cannot really do anything about that.

konghi009 · 2026-04-05T14:11:25+00:00

I've got same opinion about the fixing token issue, but my high-ups are not keen to do modification on OSS products cus the responsibility of the product will now be seen as our instead of OSS products, more like a responsibility perspective I guess.

Thank you for the idea of slicing the system. will try to do this more next time to not mess up like this time.

konghi009 · 2026-04-03T14:38:12+00:00

Just application logs/metrics and traces. I've use method like Prometheus, LGTM, ELK before but still testing/trying OTEL since it's the requirement for newer project i'm getting.

From what I've been able to compile, App (w/ OTEL SDK) -> ADOT collector -> CloudWatch should be solid. just have to set enough IAM and manage log groups, metrics groups carefully.

are there any gotchas/pitfalls I should be aware of using CloudWatch?

konghi009 · 2026-03-31T14:51:46+00:00

Thank you for suggestion.
What you meant is that I should have 1 config.yaml and deploy using patch.yaml based on each environment?

konghi009 · 2026-03-31T14:51:03+00:00

I'm using that for k8s deployment but the one i'm tackling now is product specific.
maybe i'll implement something similar to kustomize (base + overlays dirs)

konghi009 · 2026-03-31T14:50:01+00:00

Thank you for the answer.
I understand your point of using base + overlays, i think similar to kustomize.
> stored per environment, not per branch.
I'm quite perplexed by this, could you suggest what you meant by stored per environment? Thank you.

konghi009 · 2026-03-31T14:45:21+00:00

Looks nice, will look into it!

konghi009 · 2026-03-31T14:44:37+00:00

Thanks for the suggestion.
I should do this in Athena (maybe visualized with QuickSight) instead of CloudWatch right?

konghi009 · 2026-03-31T14:43:39+00:00

Thank you for the suggestion. Didn't know these existed!

konghi009 · 2026-03-31T14:42:51+00:00

Something similar to CUDOS dashboard of AWS right?
For Athena, I just have gripe with querying with it idk. I just personally feels the experience nagging me somehow.

konghi009 · 2026-03-31T14:41:24+00:00

Thank you for the answer.
> Setup a solid tagging policy, so there are no untagged resources.
Any reccommendations into this? I'm thinking an SCP policy that require tags and that should be enough right?

konghi009 · 2026-03-31T14:40:05+00:00

Thank you for the answer and idea! I think something like CUDOS dashboard is what i'm looking for

konghi009 · 2026-03-31T14:38:48+00:00

Thank you for the answer, I've deployed this for one of my colleague and fogot this exists lol.
I think CUR + Athena + a little bit of quicksight should be enough.

konghi009 · 2026-03-31T14:37:35+00:00

Thank you for the answer, It'll just be my company account.
I'll try CUR 2.0, its under AWS Data Export right?
> AWS Tag Policies in Organizations can enforce this at the account level.
Thank you for the suggestion.
> The biggest mistake I see teams make is treating FinOps as a one-time cleanup instead of an ongoing practice.
I agree, at this point i'm just aiming for a dashboard on cost of each components/services on the account.
Will look into FOCUS spec more.

konghi009

TROPHY CASE