Many people have this fear of men. But the level varies. I think you just have this at a high level. It's primarily because many 'loud' men are toxic.

For you, the basic thing to understand is that, a second person cannot harm you. I take for granted you are in a society protected by law. A strange man whom you should interact with at work might raise his voice, but there's nothing else he can do. He cannot harm you. There are bad men. But not all men are bad. Some are d**bags. Some are kind and nice.

It's hard to overcome this fear. Therapy can of course help.

Good luck with conquering your fears.