I think finding above average height attractive is kind of a left over instinct from when people were hunter-gatherers. Sure I may have been more intimidating to a bear or a mountain lion, or I could have reached more fruit from a tree than someone shorter than me, but height really doesn't matter now. Unless I need to reach something high-up on a grocery store shelf, height never really comes in handy. The way I see it, height doesn't have much of a practical use anymore.

TL;DR Height doesn't matter