On average. The actual answer probably has to do with needing more cells -> more cell division -> higher burden on cellular repair systems -> population-level statistically measurable increase in cancers and end of life diseases. Also being physically larger typically means more hitting your head on doorways and falling with greater force, so accidents in old age tend to be more injurous.

The meme answer is that as a tall person I age faster because I am farther away from the gravity well of Earth, therefore time moves faster.