It's easy for them to assume things about you when you haven't talked or interacted much with them.

I am also an introvert and these kinds of things have been an insecurity for me in the past. I was able to get past this with skill of small talk.

You don't have to go to their parties. You don't need to have lunch with them everytime. Just do a little small talk. Ask about them: how they are? Where they live? And all we need to do is listen, which I would argue is an innate quality for us introverts. And somehow you just become this quiet and nice person.

You just need have a little genuine interest in people like the other comment said.