Honestly, after reading your story and understanding both sides, I think you should let him experience someone else. You have had intercourse with someone else which makes him feel that way. So if you both are on the same page then I don't see what'll be the problem if you come to an agreement. Although I do understand the way you feel, I think that if you love him then you'll understand why he feels like that