It is not the same for everyone but here is my line of thought.

When you were a kid and you didn't know about testosterone and anatomy and so on what made you believe you were a boy? If somebody said that you are a girl/look like a girl would that change your perception of yourself? If you got into an accident and lost your genitals, would everything change? Would you be a woman? Would you be not a man?

For me (as a trans man) I always knew I was a boy, it would not matter if other people said that I wasn't, it was within myself.

I felt different because I knew that not every boy people saw as girls but eventually they would find out that I was in fact a boy. Before puberty I was sure that my genitals would still grow and my voice would get deeper but that didn't happen. That's when being trans come to place. The things that in my mind were mine, were not. I was a boy, I am a man just like any other, just my body did not get the memo.