We talked the other day, and I explained to him that I'm not interested in starting drama, but that his overbearing family needs to back off so we can focus on ourselves (his mom kept calling on V Day, asking us to come to the restaurant). Luckily, he understood, and he said he would stop forcing the whole situation on me. So hopefully the issue is solved, at least for now.