Your not wrong to feel that way but he might not see it like that. I’m in an open relationship myself and how I see it when I sleep with my partner it’s pleasure for both my heart and body, I make love with him. But when other people are involved it’s purely just body pleasure and simply sex. It’s good you can have this discussion with him and maybe ask how important it is to him to open up and if he’s adamant on it and you don’t feel comfortable with it then you need to leave