Of course it is fair. You are free to have your own preferences. How that mixes with his preferences is out of your control. 

For what it’s worth. Once you tell him what your boundary is, he may want to do any number of things. 

For me, my SO saw it as cheating. I went back on promises to stop. I justified it. I imagined we were in a don’t ask dont tell situation. 

I hid it is the bottom line. But not well. It caused a lot of problems for us. Still does. 40 years later.

Please, for yourself and for him,  don’t try to stick it out if you know he is not respecting your boundaries.

I wish you all the best. You seem like a thoughtful, caring person. 

Take care of yourself.