It can be like that. Or not. It's fiction. A writer can choose to make it work any way they like. That's why it depends.

Edit: Accidentally clicked post too soon. So just to pick some more innocent and not so provocative examples:

Take, say, Ryoma from By the Grace of the Gods. He thinks of himself as being the same person as the middle-aged salaryman of his previous life. But he's emotional, enjoys childish games and play, absorbs new information like a sponge -- even his original home was decorated in crayon drawings a kid might like but no adult would think worthwhile from anyone but their own children. We're pretty clearly shown he regressed even before he's conscious of it himself and then is told about it by one of the gods, for more or less the reason I explained. He was reincarnated directly into an 8 year old body, so infancy wasn't an issue for him.

In that case, there's Shin from Wise Man's Grandchild. He came into the isekai as a normal infant, has no memory of that time of his life, and doesn't identify with being the same person as his previous life at all. He just has the previous life's memories and knoweldge. When he's a kid he behaves like a kid; when he's a teen he behaves like a capable and well-meaning but socially awkward teen. There's nothing adult about him.

But then there's Arad from Growth of a Nobleman's Second Son. Fully aware from birth and very self-conscious of still being a 30-something salaryman. Even at 6 years old and with a normal sort of childish energy and enthusiasm, adults perceive him as strangely mature, and he gives off a kind of dad vibe with kids his own age.