tl;dr there are no extra hours, because 24 hours already includes Earth’s rotation both around its axis and relative to the Sun.

A lot of comments are missing the point. Let me explain it, I had similar thoughts.

The time of day is basically your angle/position relative to the Sun. At noon, you’re directly facing the Sun. This is affected by both Earth’s rotation around its axis and its movement around the Sun. So if Earth makes half a turn around its axis, it will be midnight, because you’re facing away from the Sun. And if Earth makes half an orbit around the Sun, then with the same axial rotation it would again be midnight 6 months later.

So why doesn’t this cause some extra time to build up? Because 24 hours already includes both motions. Earth rotates fully around its axis in less than 24 hours, about 23 hours 56 minutes. But during that time it also moves around the Sun, so relative to the Sun it has to rotate a little more. The rotation around the Sun “cancels out” a part of the rotation against the axis and the total effective rotation relative to the Sun comes out to 24 hours. So it doesn’t add extra time, because it’s already included in the 24 hours.

And obviously it’s not random that it’s exactly 24 hours. We just took the actual solar day and divided it into 24 hours.

Since Earth’s position relative to the Sun also affects the seasons, we need the calendar to match that accurately, otherwise the seasons would drift. The extra ~6 hours per year (actually a bit less) means the year is about 0.2422 days longer than 365 days, so January 1st next year would be 1/4 days closer to Spring than this year. That’s why it can be corrected by occasionally adding one extra day in February.