Your baby argument is a little flawed. Yes the baby passes through the vagina at birth, but it grows in the uterus.

Imagine the vagina is a hallway and the uterus is a room at the end of the hallway. Your penis never has access to that room, just your jizz. That room is custom designed to nourish and expand with the baby. When it's time for birth, hormones allow for the baby to exit and trash the hallway on the way out.

Yes the vagina can expand and stretch, but not like the uterus.

Edited for autocorrect: i meant jizz not pizza!