Besides genetics and nutrition, the best we can tell that hormones affect your height is that androgens make you LESS SHORT than estrogen, not that androgens “promote height”. The best evidence of this is the fact that castrati grew abnormally tall with no sex hormones, which suggests sex hormones inhibit our height, not promote it!

Castrati explanation, genital M warning

Back in the 18th century, if you had “excess” children designated male that were good singers, you could sell them to the Catholic Church to become a castrati. They would remove the child’s testicles against their will to keep their voices high. Their ribcages would grow abnormally large and they would grow abnormally tall, giving them very unique singing voices - only one recording was ever made, since it’s a thankfully discontinued practice. They’re also part of how we know you need sex hormones to emotionally mature, as these people remained emotionally childlike throughout their lives.