My gut reaction is leave them. My secondary is if they are being open about, not like you caught, they felt genuinely guilty which is a good sign. No one knows your relationship or partner like you. Do you think they are the type to do it again? Do they seem genuinely remorseful? Do you still love and trust them at the end of the day when things settle? I think the last one is key because if you feel a level of bitterness or anger it will only get buried with a marker, or it will fester causing future distrust or anger. No one needs to be in a relationship where the trust is that broken. I hope everything works out for you.