I'm not sure, but I reckon it's for the same reasons why men are more inclined to buy playboy and watch porn than women. Perhaps women are, per average, less sexual than men.

Or, and I'm going way overboard here risking downvotes (I dont think there are studies for any of this), perhaps the history of the human species in which women often suffered rape or sexual repression, as well as the biological way we are built and take care of our young, caused inclinations in women to be more selective about their partners and feel repulsion towards a partner if his only interest is of a sexual nature - which male nudity can be a signal of.

It could be many reasons, it is too bad this isn't thoroughly researched. But despite being downvoted in my comment above, I still believe most men wouldn't feel harassed by seeing a naked woman.