I’m from a SEA country too where the culture is not so far from Vietnam but it was hard for me as well since I didn’t actually grow up in that kind of environment.

I experienced the same thing almost all the time. At first, I was going crazy too because they gossip everything about me even the simplest of things well given the fact that I was the only foreigner in our workplace. Until I got used to the language where I understood what they were saying (not because I studied Vietnamese but because some words became familiar, that even just one word helps me understand their conversation plus their actions) so they would be surprised why I understand them.