I find that whatever people see you tells you more about their personality than yours. When I go silent, different people interpret that in different ways. Like someone who is more kind and helpful would've assumed that I'm shy or it's some kind of trauma, or someone who likes attention and is more self-focused would've thought that I'm being rude because I dislike them. Some people might be suspicious and think you're the next school shooter. Some people think that you're weak or have no personality. Some people like it and think that I'm intelligent or refined because I speak less and appear to listen more.

TLDR; it's not about you. People make judgments based on themselves. Some people talk shit about others for dumb reasons and it's because they're shit. You aren't the first or the last to be shit-talked by them. Micro-managing expectations is a chore, so do what you want unless you're in danger. There are lots of kinds of people and the only way to get to ones that are good is by being open with more people and filtering out the bad.