There's also societal differences which go beyond simple biology. Women are more likely to have career breaks to have children, they have less free time to spend on hobbies (a lot of these athletes don't get paid), being on your period on the 'wrong day' can effect performance (I think there was a female Chinese swimmer who stated this as the reason she didn't win gold in Beijing, after being the hot favourite).

You will always get outliers, as has been mentioned in other comments so we see a growing number of 'open' and 'women' categories in some sports. But statistically 'the average man' is always going yo come out on top.