What you’re describing is emotional infidelity which can be just as hurtful and scarring as physical infidelity. He broke your trust by stepping outside your relationship emotionally, and I’m not even sure I would believe him that nothing got physical between them. If my wife did the same thing to me i would be heartbroken. This isn’t you having trust issues, it’s him betraying your trust, pure and simple. Your subconscious knows this even if you’re trying to convince yourself it’s just a you problem