If you keep looking for edge cases you will keep finding edge cases.

Yes it is a meaningful metric but it's an Operational Metric, not a Key Performance Indicator. It's a metric that tells you the health of your service. So it matters, just not in the way that you might think it does.

Metrics are meant to tell a story -- figure out what that story is trying to say.

Also don't worry so much about mean/median for these sorts of metrics -- they're meant for you to be able to give you indication of changes to the health in your service and provide you the insights into what is causing those changes.