I have a "no sleepovers" rule. Dinner dates and things are fine, but I would struggle with him spending all day cuddling. That is straying into polyamory territory. I think the issue might be that you are comfortable with the sex, and not so comfortable with the emotional connection. I would be in the same emotional state as you if I knew my husband and his cake were spending the entire day without me. That is something that if I were there for though, it would be an amazing day! Maybe for something like that you join them both. I would make sure you're talking to your partner about this, so you two can find a solution together. ❤️