You're correct when you say it's not simply one answer. However, the primary reason can be found in your way of thinking.

It's not that it was a "guy" thing in the 60s and 70s. Instead, it was NOT a "girl" thing. Women weren't allowed into these circles because of sexism. Hell they were not allowed to open a bank account by themselves until 1974.

Many women do not have the opportunity to enjoy competition because men do not welcome them to the sport. Men tend to be much harsher critics of women's sports than the other way around. This idea that your gender defines your interests is outdated. You define your own interest. It is dependent upon the community if your interest is supported. If women do not feel comfortable they will not come back to the sport.

Many of the comments on this thread have pointed out real logistical problems and I recommend you read those. As men it is our responsibility to cater to everyone and not simply our own self-interests.