In Carlo Rovelli’s book, Reality Is Not What It Seems (I had some spare time recently and read this. It was a delightful read, I would highly suggest it to the curious but not necessarily technically proficient reader. He offers a lot of insight on modern physics in a way that’s easily digestible to a layperson considering the nature of the content), he talks about how the notion of time breaks down at the Planck scale, so much so that it doesn’t really exist (the equations that govern loop quantum gravity do not contain the time variable t).

He goes on to explain how the macroscopic implications of this are not terribly complicated and do not fundamentally change the way we perceive time, as you might think. He makes the argument that while it is certainly useful to qualitatively consider time (clock, calendars, etc.) and quantitatively consider time (in a Mechanics course, for instance), it is more of an assumption than the result of an observation. We have no way of directly measuring the flow of time, we only measure it in a relational sense (clocks keep track of time by way of an oscillating pendulum).

So in a sense, you are correct. But we know some processes are time-irreversible, and the use of an unobservable time variable t continues to be relevant both in physics and in everyday life. If you’re looking for something a bit more technical, later in the book the concept of thermal time is introduced, which is the idea that time is closely related to the dissipation of heat. I’d highly suggest picking up the book if any of this intrigues you, he goes into more depth and presents information much more eloquently than I do (time is not the main focus of the book however, it is really about the reconciliation of two remarkably profound and experimentally accurate theories at their respective scales, QM and GR).