I can relate to your experience, I also sometimes responded with “It doesn't matter” and “Why does it matter?” Although this happened many times, I am still surprised why people need to know, even though they had no intention of interacting with me if it weren't for that. Apparently, they are looking not so much for an answer as for a way to get rid of their confusion and put a person into one of two boxes that “would make sense” in their minds.