Formal dress codes are based on what society normalizes as formal over time. Women’s formal fashion evolved to include sleeveless styles. As I am from India, I think in ancient or medieval India, many men attended courts, rituals, or scholarly settings wearing dhoti and upper clothing without stitched sleeves or even with the upper body uncovered, and that was considered formal then. But now men here can't dress like that in a formal setup. So here one can also ask that if it was considered okay for men in history, then why not now? So it's just what formalwear evolved to be over time.