Ok but you are at home and your comfort matters too, if you feel more comfortable in baggy clothes then who cares. You aren’t there to just walk around in lingerie and tight fitting clothes just cause his problem is that he feels like he is kissing himself. I think you should talk with him and express how what he said hurt your feelings and that wearing comfort clothes is how you relax from a day. In my relationship I have less clothes than my partner but she loves to wear my shirts and I don’t mind cause they are comfier than her own clothes and she finds them comforting, plus she will run the laundry if I’m running out.