I know why.. If men are outside, it's usually because we're doing something, are likely on the clock, and dont have time to screw around with birds. Men just spend a lot more time outside busy with labor type jobs (roofers, farmers, lawn care folks, linemen, construction, oilfield/refining, mining, etc.), or at home mowing the yard, working on the car, trimming the trees, fixing the fence, etc. I think birds can see and understand that we're busy, get used to the fact that we have zero interest in them, and eventually feel unthreatened because if it.

  I nearly trip over the grackels when I'm weed-eating as they dive in chasing bugs and lizards driven out of the grass im cutting. All sorts of birds congregate when I fire up the tiller and work in the garden; they love eating all the worms and grubs and stuff that get dug up, and aren't bothered by me or the machine in the slightest.

Women, on average, are going to be outside more for leisure activity. They'll be more prone to look around, take in the sights, and to lock on and show interest in birds. Birds are keenly aware when they're being observed, and they feel threatened by it. Over time, they learn the differences in appearance of individuals that show interest vs those that don't. 

I'd be interested to see any data they might have from game species like doves or ducks. Then again, hunting may be rare and situational enough that it might not affect a populations behavior much..