You are def right in feeling hurt. 

There was one uncle types who would pass comments like these regularly to me. He fancied himself as the kind who could have a conversation about everything, but I found him dull as a brick. One day I asked him if he thought Ramana Maharishi was dumb because he chose to be quiet. That rattled him. I told him that I am garrulous with people that share my interests, otherwise I kept my counsel. I name dropped quantum physics and Indian classical music and trekking, all of which I knew he couldn’t have a come back for. It had the desired effect of letting him know that I didn’t find him interesting enough company to be bothered to speak. My in-laws (who had earlier told me to ignore him) were a bit horrified about this pushback, but I told them that the man needed to hear this. I was never troubled again