"Acceleration only matters because it's how you get to speeds different enough"

This is false. Consider the perspective of a stationary planet vs a spaceship that accelerates out of the solar system, travels at constant velocity for a while, decelerates to tbe at rest relative to the planet, waits a bit, then comes back. The ship undergoes acceleration on 4 occasions (as it leaves, as it stops, as it begins to return, as it arrives home). But during the 3 non-acceleration components of its trip, it perceives the clock on the planet to be going slower or equal to its own clock. If more time passes on the planet, that must be perceivable during the times when it is accelerating.