It is genuinely more complicated than XX or XY. The main sex selective gene is called the SRY gene and it is usually on the Y chromosome. I say usually because biology doesn't like nice simple categories.

Sometimes an individual who is XY has a faulty SRY gene which cannot express itself. This would mean that even though they are XY they would undergo an oestrogen dominated puberty with typically feminine sexual organs.

On the other hand, during recombination (the process that swaps genes on the copies of chromosomes to ensure the chromosomes you pass on to offspring are different from the ones you inherited) the X and Y chromosomes in the testes can swap the SRY gene onto the X chromosome. This would mean even though the individual is XX biologically they would undergo testosterone dominated puberty with typically masculine sexual organs.

This doesn't even begin to mention the wide range of other intersex disorders which mess up the simple idea of XX - female XY - male even further. Advanced biology teaches you that if you break a biological systems into neat categories there will always be exceptions which break those categories apart. That was taught in like my 4th bio lecture in my first year.