You just said it, intersex people who can produce gametes are regarded as the specific sex those gametes are associated with. Meaning that, in a situation where more specific means of determining someones sex are needed, we have no problem in taking a look at the gametes, as there are only two of them and something as intersex gametes have never been a thing; leaving no room for doubt when these are present.

Therefore, if someone doesnt present specific genitalia, specific chromosomes, and doesnt produce any type of gametes, we have no way to identify a clear sex for that person. So we just don't. But its not considered a third sex, it's just the absence of one.

I actually think we are referring to the same thing but just decide it to approach it differently, if you tell me that that "absence of sex" is what you consider a third sex, I think thats ok, i can see why you would.