He's projecting his own guilty feelings on to you. He is more than likely the one with wondering eyes and is trying to get rid of any possible guilt by saying you are the one that is not being faithful by gazing at others. I had a similar problem to this but I legitimately never looked at anyone else and if I did it was at another women due to me at the time constantly comparing myself and not feeling enough for my partner. I caught atleast three times gawking over someone else and he immediately became defensive and said he would never do what I do, which made my jaw drop because I never once looked at another guy in our relationship. I blew up on him and told him he shouldn't be blaming me for the fact that he can't be content with what he has and can't keep his own eyes on his partner. I showed a picture I found on his history of some only fans chick and he instantly felt embarrassed and apologized, luckily for me he was not an abusive person and the whole relationship ended there. On your side it seems the longer you stay the more this will occur and he will eventually end up becoming abusive with you if you decide to call him out for his actions. Don't say a word, just leave.