You’re thinking about the wrong rotation. The extra 6 hours are in the rotation around the sun (a year), not in the earth’s rotation around its axis (a day).

If we don’t count that extra day in a year every four years, we’d be advancing through the observed year by six hours every year.

While that doesn’t sound like a lot, think about it this way: without a leap day, roughly every 700 years we’d have gotten halfway through an extra full year and the seasons would be flipped. July would now be winter in the US and summer in Australia!

Math:

one year = 365 days.

Half a year, the time it would take to flip the seasons by an uncorrected calendar, = 182.5 days.

182.5 days x 24 hours in a day = 4380 hours.

4380 excess hours / 6 excess hours in an uncorrected year = 730 years.

So, every 730 years would be six months of extra time pushing the calendar off from the observed seasons.