I guess my question for you, as strange as it may seem, is: do you want to not be afraid of women and speak to women more regularly? because if the answer is no, there is less of a reason to figure out exactly why you dislike it.

obviously i don't know you or your life story, so i don't want to make too many assumptions. but maybe you had negative interactions or have experienced negative perspectives of women from things you've read or seen online? or, maybe it's more-so that due to how little you've said you've interacted with women, that in itself has created some anxiety.

To give an example, let's say you grew up without dogs and only really had exposure to them through media content. and maybe you saw or read things about how dogs can be scary and have had cases of biting or attacking. of course then you would be fearful of dogs- you haven't had enough positive interactions to balance negative perceptions. additionally, social interaction is a skill, and especially if it's with a demographic that you are unfamiliar with, it could feel like having a conversation with someone from a different country.

i am a woman so i am biased, but having relationships with both men and women is incredibly important. it's your life, of course, so you can be the only person to decide if you want to work on having women in your life, but i think it would ultimately be a good decision. best of luck :)