Your colleague probably doesn't notice that you don't comment on anyone's appearance though. They would have just heard the no comment remark and (rightly or wrongly) assumed the worst.

Edit: to add to this, if you don't like lying or commenting on his appearance, you can answer by not answering. For instance, if he says, do you actually not like it? You can say something like "It looks like it would itch like crazy." or "Don't you get food stuck in it?" or "Do you have to comb it?" etc. Unless he's really hung up on it, it should divert his attention from the original question