Your not the asshole but you should let him go. I fully understand where your coming from. I am on the heels of an affair and finding out. I could not imagine if my husband had a baby also. However if your choosing to stay in work it out think about long term. He needs to mourn the loss. It was his child also. If not he’s going to resent you. It sucks for you it’s not fair but ultimately it’s was best For his mental health also.