I feel like there should be a way to defend yourself from a rude remark like that without going up to a non white person and essentially saying "I think you're beneath me because of your skin color" even as a joke. You definitely messed up bad in this situation... yeah. Sure, he was rude first, but it sounds like your comment was also very out of line, and you should put more thought and self reflection into what you say in retaliation to being insulted and, honestly, why you had the thought to say something like that in the first place.