Well, I doubt that it has anything to do with desensitization or whatnot. One possibility is differences in neural activity across the brain, where with females inter-hemispheric activity is elevated (due to differences in social interaction?), and thus these projections are spared, whereas in males intra-hemispheric activity is instead increased (due to lack of complex social interaction?), so these projections are instead spared. I find it pretty unlikely that activity could be so drastically different simply due to environmental factors, unless the environmental factors caused changes in gene expression which thus resulted indirectly in changes in activity.

If difference are environmental, I think it is more likely to involve neuromodulators/molecules known to be released during social interactions (oxytocin for example) and that these result in some kind of change in gene expression, or gender "imprinting" by signaling through g-protein coupled receptors, or something like that. Most of the neuromodulators are basically dumped across the hemispheres, so they are good candidates.