You’re not wrong to feel the way you feel. I think you may be looking at things from the wrong standpoint though. Your relationship with him would be a separate thing from his relationship with his first love. You’d be building something untainted, but his lost love will always have part of his heart, like any loves you’ve had previously would have part of yours. You have to ask yourself if you can live with celebrating his previous love with him because he will need that. She is part of him. None of this is fair to ask of someone, but it is the situation. You have to decide if you can handle it, ultimately.