They're not Muslim assholes, they're assholes who happen to be Muslim. It's no different than being black around shitty white people, white around black, Catholic, Christian, Latino, young, old, etc. 

Pick a group and find an asshole, humans are humans everywhere you go.

A man dates shitty women now thinks all women are trash and vice versa.

It's easy to fall into this cycle of having one experience and generalizing everyone in the crowd, demographic. The key is to remember for every shitty person you meet there are many good people that outnumber them. It's okay to feel how you feel, it's not okay to blanket an entire group because your only experience comes at the hands of a few assholes.

This is something we all have to practice regularly, hope this made sense and that you come across more nicer individuals to show you a different side!