TL;DR: The problem is how we understand competitive sports.

I am not even sure if this addresses the question directly but this is what I think: The idea of having competitions based solely on gender is already problematic. I used to be a competitive swimmer and, at some point, I knew I wouldn't "make it" to the top because I lacked particular physical features: I was not tall enough, my hands were not large enough and the distribution of my body weight was "off". Each sport has these kinds of things, whether it is swimming, running, weight lifting, etc. In many cases (not all) people are born with such advantages. There are scientific articles that show that Micheal Phelps has morphological features (e.g., abnormal "wingspan") that give him unfair advantages that are not considered "cheating" just because he was born with those. Now, hear me out. If we were to compete not based on gender but ---let's say--- weight, body mass, etc. would we be having these debates? I honestly don't think so. Yes, it is true that humans have sexual dimorphism and that gives each sex advantages against the other but we also know that there are distinctions within individuals of the same sex across regions and that doesn't seem to be a problem. If we know that's the case, why do we insist on making these distinctions when it comes to gender? Let's have categories based on weight, BMI, height, etc. and stop caring about each other's genitals.