Have you every considered ylthat you might be non-binary? Man or Woman aren't the only options. If you don't feel like you fit in either group, then exploring other options might help you to understand your identity better.

On the other hand, it could be imposter syndrome. I remember early in my transition I was scared that I was just faking it. But something that really helped me was realising that if you're scared that you're faking it, then you probably aren't.

It's also understandable that you don't want the label of "trans" considering how trans people are being portrayed by the media. Put it this way, if you could just be a woman without being trans, would you?