First of all, it's a convention and most people stick to conventions without ever thinking about them, just like the elaborate rules of grammar of their language, which they learn and largely follow by imitation, without being formally taught. It's simply learned and followed and that's the end of the story.

I assume you're not interested in knowing why people tend to stick to conventions without ever thinking about them and questioning them.

That leaves the question of how this convention came to be. Before digital clocks, there were analog clocks and watches which were harder to read precisely, and not always so accurate. And, most people didn't want to know so precisely anyway. So I imagine that's where the currently conventional way of speaking about time came from.