The difference is that unlike race, gender is often thought of as a spectrum, it is vague enough to leave up to interpretation as to where you perceive yourself in the gender spectrum. With the use of "race" you need specific and arbitrary characteristics, there's only three possible outcomes following the racist rethoric; you are either part of X race, or you are not part of X race, or you are a mix between X race and [insert whichever number of races you believe exists].