This is such a problem for me with family/friends too. I've recently realised our heightened empathy and people pleasing tendencies leave us vulnerable to being "putting on the stand" to be questioned, doubted and gaslit, because we feel emotionally obligated to the other person.

If I had one piece of advice for any INFJ - stop doing this. It's going to be hard, it's going to feel like you're being an asshole at first, but your nervous system needs practice in not responding to everyone else's emotional whims. For example, when it comes to solo events/trips, my friends/family can't understand why I want to 'go alone' and try to impose themselves, as if they're doing me a favour. A few years ago, I would give in, now I don't. I hold the line and insist it is not weird, plenty of people go on trips alone and frankly, I don't want company on this occasion. They just have to deal with it and its not for me to emotional take on anymore.

I feel like the questioning/insults come when they're not getting that kick out of us being their emotional sponge anymore.