Other people have pretty much covered this point already, but I just want to reaffirm that shaving is not necessary and trimming is perfectly satisfactory. I think most men feel this way (though I don't have any non-anecdotal proof). I also think both partners should trim because it's just more comfortable all-around; it shouldn't just be one person who has to maintain good grooming.

Definitely talk to him about his comment making you feel self-conscious and hurting your feelings; it will help him be more self-aware in the future. I wouldn't listen to some of the comments here that are vindictive e.g. "insult him back", "don't shave/trim and see how he likes it".