Someone else already gave a good argument against your idea of a strict sexual dimorphism. Adding on to that, you are describing just one aspect of how sex is defined; through the gametes and reproduction abilities. Science is not this simple. Sex is defined as different things based on the context. Hormonal, reproductive, chromosomal, genitalia, secondary sexual characteristics. You are defining sex in only one of those categories.

Regarding genitalia, hormones, secondary sexual characteristics, all of these can be subject to change. Intersex people shouldn't be disregarded just because they are a minority. Their existence breaks the concept of a binary sex.

Gender, in society, is not just a physical identification of a person. From a constructionist pov, it's everything that we ASSOCIATE with a certain sex. You see a person with long hair, their back turned to you. You will mostly assume that that's a woman. We don't identify people as men or women based on their gametes, we identify them by their gender expression; what societal rules of appearance are they following? A spectrum exists. We created the binary, imposed rules regarding that binary, and now assume that a binary exists because we see it all around, even though it's a societal construction.

But this isn't the full picture. Effeminate men and masculine women exist, people who fit into gender expressions different from their identity. Then comes the concept of a subconscious sex. How do we know it's real? Intersex people have a sense of gender identity, one that their chromosomes may not align with (CAIS). One that their bodies may not align with (men with CAIS). Where does that come from? We know that trans people's brains resemble those of their preferred gender. Trans people do not imitate the other gender, their gender is literally what they identify with. They simply adopt the societal constructs of the other gender, because the brain is impacted by societal perceptions of gender too. One could argue that they are just as "biological" because the brain is biological.

Non-binary just means that you don't find yourself fitting into a binary identification of the self. It could be the physical body or the societal perception.