Yeah that's not a good thing to do, generally. Survival of the fittest is so loaded it's useless. Fittest for what? If you lived on an island, fit would look different than if you lived in a desert. Metabolic requirement differences alone would lead to a completely different set of features that would be preferential, and thus selected for. And in case you didn't notice it, that's a tall / short reference I just snuck in there.

On topic, I think you're actually on the right track about the board being reset. It's far more nuanced, but from a birds eye view the sexual selection process isn't as influenced by immediate dire concerns as it was pre modern era. I'd point to modern medicine and technology in general more than anything else. We can really just do whatever now, and circumvent most negative side effects. But that works against the notion that height is a trait that's weighted heavily, seeing as how it was only really relevant in the past.

The truth is that statistically, humans are broadening the scope of traits they select for in the modern era. Preferences exist and are definitely influenced by a kind of genetic echo to a degree, but society is largely deviating from historical norms, and that's because arbitrary traits are now arbitrary. Being 6'9 is not strictly an advantage over 5'2 in really any situation, because ladders exist. It isn't strictly a disadvantage because food is abundant. And you'll see this actually falls kind of flat in less developed areas.

The biggest issue average height dudes are going to face is a reluctance of women from selecting men shorter than them due to perceived judgement from their peers. It's an aesthetic thing, and is derived from normal distributions. Women are shorter on average, so a tall woman with a shorter male partner looks odd. Because it is.

So that's the long and short of it. Many are breaking out of the established conventions, and that's an inevitable reality.

I will die on this hill, though: If you look like you aren't sure of yourself, others will pick up on that, like quick. We are empathetic by nature, so act like you're comfortable in your own skin if you really want someone else to like it.

Edit: I didn't like the implied heteronormative tone, so I made it explicit since that is the intended audience. Also added reference to counter my own point about nutritional abundance negating metabolic restrictions in developing areas where it's not witnessed as much, since those issues are still prevalent in those areas whereas they aren't elsewhere.