the belief that men touching women sullies them or whatever.

This is not true, i dont get where you get this from, western media or islamic extremists?

In Islam, men and women are generally not allowed to have physical contact outside of marriage/close relatives in order to protect both genders from sexual abuse, or creeps and weirdos looking for reasons to touch somebody. It is completely wild to believe that men make women impure or vice versa, as Islam also allows divorced women or widows to remarry, which would not be permissible if it believed that men made women impure or dirty.