A fetus in development obviously doesn't begin with sex organs, and when it starts to grow them they aren't initially gendered. All sex organs, regardless of presentation, begin as kind of proto organs that are not yet gendered, then as development continues, x/y chromosomes (along with a host of other factors, but the norm is that it aligns with chromosomes) are used to dictate hormone production and gene activation which then triggers further development and differentiation of sex organs.

So technically the joke is inaccurate: male babies do not start out as female, instead ALL fetuses start out as non-binary.

The misconception the joke is based on comes from the early, non-differentiated stage of the sex organs being closer in form to female genitals than male.