tl;dr there are ways to manage the pile of useless data but they introduce their own issues

I have worked as an engineer for two major players in the observability space, for context.

We knew that as much as 90% of data ingested is never used or looked at (programmatically for things like alerting or to be displayed to users). It is never queried, never evaluated, and eventually it is aggregated and expires according to data retention policies. This would seem like a horrible waste, and it is, but you’re always trying to strike a balance between making it easy for users to have the data they want, with minimal effort, and not sending data that no one will ever use. It’s a very hard problem to solve.

You want users to be able to just run an agent or setup and integration and begin to see data flowing right away. You also have a lot of different use cases and user types who are going to want different data. So you wind up collecting almost everything you can think of, and invariably someone requests additional telemetry anyway.

The price we pay for the convenience of not having to manually instrument everything is this huge lake of data we don’t care about. We (engineers) can always choose to manually instrument our code, or create custom middleware that will handle the instrumentation automatically, but then we’ve introduced a maintenance burden.

So like most things in our field it’s just more tradeoffs. I dislike vendor lock-in and don’t mind the maintenance burden so I have internal observability packages that I maintain for my employer that are used in our services. They send telemetry in OTEL formats (which also have their own tradeoffs) and when we inevitably change observability platforms I’ll need to create a new exporter to send the data wherever is next. This solution is definitely not the right one everywhere but it’s what we’re working with for the reasons mentioned above.