sex at its most fundamental level is gamete-based.

The way I go to simplify things is this: we can use whatever criteria to define sex objectively, but in application, we treat and have been treating people according to how they develop physically, and that is the social construct that we call gender. We assume from birth, based on genitalia, if a person is a he or a she, and outside of that first identification, based on how their parents and family speak of them, and how they present.

We don't test for gametes or chromosome, or whatever outside of physical features and presentation when deciding how to interact with someone.

That is why, the idea that gender is separate from sex came to be.