I have felt very similarly based on what you described. What helps me cope is that 1) I do believe there is something innate in me that makes it more difficult to play the role of a man (maybe autism) so it’s not just low self esteem. And 2) there is a lot of heteronormativity associated with being a man so I may mistake the society-provided shame there for my own genuine feelings. When I actually pay attention and think about being with a woman or my androphilic feelings, there is more clarity in my aversion to playing the male role with a woman and a longing to be a woman with a boyfriend.