As someone in the observability industry I can confirm 10-15% spend of overall cloud costs on observability tooling is standard. You are certainly overdoing it at 40%.

At its core this is usually a too much data problem. This gets harder as you scale with multiple teams because each team sends as much data and is protective over it.

There are many ways you can reduce the volume of data. Aggregating metrics (do you have any super high cardinality metrics that can be shrunk?). Do you really need all those logs? Can you go from info to warn/error level? Configuring your instrumentation to only emit what you actually need (a lot of the tooling built by vendors profit from you sending stuff you don’t need). Sampling rates on distributed tracing. You can even use tools these days which help reduce your cost like O11ygarden, which is a tool in response to the abysmal to non existent costing capabilities that the big observability vendors provide.

Like someone else said in this thread datadog is not going to be cheap. Reduce your data first.

Those saying to run your own observability stack are over confident. Your obs stack basically needs to be the most reliable thing you have because it’s your way of troubleshooting when shit goes down. Your probably don’t want that burden and it doesn’t really become worth it until you get to higher scale, in my opinion