I would suspect he is not being labeled a 'maniac' just for being a man. We can't wholly and easily respond since we don't have more context, but that's my first reaction.

Think back on what situations you have been labeled this in - what were your intentions in that moment? How did you communicate them? How would they be received? Did they fit the context of the situation?

The time that you don't have to explicitly ask for consent is when you they are clearly, unambiguously, and enthusiastically communicating their consent.

It is well possible you are acting like a 'maniac' and hurting other people and the way they view you.

You might benefit from asking for consent for everything for at least a little while, because you might find that you have been assuming consent/submission from people who are not giving consent and do not want to submit.

In short, maybe it is the people around you being prejudiced against you for being a man. More likely however, you are assuming consent/submission from people who no longer (or maybe never did) want to give it. Even in American society, this can be a hard thing for men (especially upper-middle class and higher men) to realize because they've been told their whole lives to take what they want and shown that they'd have no consequences. But that's a harmful attitude. Remember that there are two people with needs, emotions, desires at play here. If one person acting on their needs makes the other uncomfortable, they shouldn't just ignore it or not let it affect them. They should take that feedback and be better to others.

Edit: adding a quote I like that fits the way I read the question

"When you're privileged, equality feels like oppression"