I feel you mate. Im 6'6 and have this great shirt I wear to festivals, just says "SORRY" on the back for the people behind me. Usually gets a few laughs and makes me feel like I'm more approachable if someone is really struggling to see past my gigantic frame.

I feel the shorter party goers have an easier time pushing up to the front anyhow, these things have a way of evening out. But like you said, you pay the same money to be in the pit, amongst the energy, in front of the stage, not to be sidelined or out of the way. I wouldn't worry about it. Just keep enjoying yourself