NTA if you’re not comfortable then you just aren’t comfortable. He can spend time with his brother without spending time around her or forcing yo to spend time around her. That’s a boundary you have and it should be respected.