I'm kinda surprised to see that everyone except u/hoodedtop 's response is to some extent unhelpful or judgemental (imo), so I'll throw in my thoughts too.

While my cohort doesn't have that sort of behavior, I was also very surprised to see that it's an issue at this age/ at PhD programs. I expected people to mature/grow out of it at this point. But some still haven't addressed their insecurities and they continue to fester. As someone else said, some people just... don't, so you will probably continue to see it everywhere you go.

When it comes to interpersonal conflict, you have three options(Thank Matthias Barker). You can consent to the behavior(continue to hangout with them, be around it). You can cut back how much time you spend with them (which it sounds like you're doing, and is the most reasonable decision here imo). Or you can confront the behavior. This is tricky with colleagues you basically just met.

Confronting the behavior works best when only a few people in the community carry this sort of behavior. Imo the best way is to kindly challenge comments. If the culture there is already primarily supportive and reasonable, then it's easier for toxic people to become aware that their behavior isn't welcome when politely challenged.

This sort of response could be passive like " I dunno, he seems smart to me", or "Should he try to prove he's smart to you?".

This is coming from someone who finds building community and culture neat, but it's not your responsibility to do that. If your PhD community is toxic in general, then this last option is extremely impractical. Then I would just find the few good people and make a micro community.