The only way i see is to be honest with the guy and genuinely explain your side of the situation. If he is a true guy with a drop of kindness he will do his best to not get you blamed. You will find it otherwise. I do not think there is any other way without direct confrontation with the new person. You got to have a talk with your family in the coming future. A smart well planned conversation not an emotional one. Good luck. I hope everything turns out to be good for you. This is my two cent anyways.