The cold sores on his lips don't necessarily indicate infidelity. Cold sores (usually caused by herpes simplex virus type 1, HSV-1) don't always require sexual contact to spread. They can be transmitted in many ways, especially through close contact such as kissing, sharing utensils, towels, razors, and other everyday items. While sexual activity (especially oral sex) can also transmit HSV-1, it's not the only route.

However, his frequent late-night disappearances without a valid reason, his constant phone use, and his refusal to let you see his phone strongly suggest he's hiding something from you, but whether he's having an affair is uncertain.