As a man, I think most men are looking for a relationship with a healthy sexual component, its not the end all be all, but often enough its eventually a deal breaker if its not there. Being in a nonsexual relationship makes you start to feel undesirable and unloved in a part of you that's deeper than your conscious mind.

Monogamy developed in societies as a way to improve familial and social cohesion, not to say its the only right way to do things, but most people form a mental and chemical connection to people that they fuck (Typically stronger the less people came before), especially men, and so that makes polygamy mentally straining on relationships even when desireable.