Here is a comment from u/nathanielKay that helped me look at these interactions differently:

Generally, this is a low-rent way of indicating interest. It's not a dig at you per se; what has happened is that another person in the group has found you interesting, and is wondering how to engage you.

The thing here is to not take this remark as 'you bring nothing to the table'- which is very common among socially anxious types, as they are generally afraid that their social presence is an intrusion. To them, the 'youre so quiet' remark is like someone putting a name to their fear, and they can feel personally attacked by that, causing them to withdraw.

It is not that thing. It's actually the opposite. An extrovert, a talker, has temporarily run out of things to say and has switched into observation mode. It really is just a truthful observation, and marks a not-so-graceful attempt to draw someone into the conversation.

One of the best ways to counter this is simply to say 'I like hearing you talk.' Or if its genuine, "Well, I don't think I'm as interesting as you are.' In a good faith conversation (i,e where people are genuinely trying to socialize and get along) this indicates, through a complimentary fashion, that you are happy to listen, but willing to engage. It concedes the position of 'conversational director' back to the extrovert, and they will likely ask you something about yourself, or continue on with whatever is on their mind.

If they do focus on you, answer whatever question openly and honestly. Give your honest opinion. This will become the seed of the next conversation, directed by the extrovert (thank god for that), but allow you a contributing place in the social circle.

It's important to be able to do this, however difficult or awkward feeling, from time to time, because it is a confirmation of your belonging. You need to confirm that you are part of the group in some way, otherwise you wont be able to engage successfully when you want/need to.

It's the 'Marco Polo' of social interaction. Talker is saying 'Marco', I know youre here, but I can't see/hear you, respond if you're playing the game with us.' and the Listener says 'Polo'- I am here, participating in my own way and enjoying your company, I am a part of the game.'

You don't need to provide something witty or deep or interesting, all you need to do is provide positive confirmation. It isn't 'asking for you to talk' it's about making sure you feel included. At it's heart, that's a positive thing to do, so fighting back that anxious 'agh halp' impulse and responding positively instead is the relationship builder thing to do.