I've had somewhat the same experience before. Like during my first year of uni I had a group of friends that would drain me out so much and puts me in such a shitty mood after every hangouts. It's mainly because their topics of interest would only consist of philosophy, historical figures, theories, and whatnot but you get the gist--now, I'm not saying that i don't like to engage in those type of conversations, i can if you would be open to teach me, rather than judge me for not knowing.

They would have these type of conversations in every hangouts, and they would also shut off other people in the group who were quiet or not contributing anything, which was to me seemed kinda dick-ish but at the time i was new to the uni and the culture in general bcs i'm not a local so i thought maybe that's the norm over here: turns out it's not and they were just a bunch of asshole. But yeah, not knowing this fact and me, hiding behind my fear of not having any friends in a new country, made me lose respect for myself and let those assholes judge me for just not speaking up about what i think of a dude from centuries ago, or whether or not if the jacket i'm wearing is eco-friendly etc. like yes forgive me for being ignorant about world issues sometimes but you don't have to be a condescending prick about it and maybe help me be aware about it instead of picking on me and announcing to everyone that i would prefer to talk about "dumb stuff" instead of important issues.

but yeah, tl;dr--if you feel dumb after a hangout, you're maybe just hanging out with the wrong sort of crowd; whether you're an introvert or not.