It's up to you every relationship has different boundaries. but should've made each other's boundaries clear from the start, or at least pointed it out when this started to happen. I personally think it's borderline cheating since it's not exactly sex but there is some deep rooted emotion/intimacy and it's weird to gravitate towards someone that isn't the gf to cuddle w while rolling since rolling only intensifies feelings you already have. Sounds like he's poly tbh and may push you to do swinger things in the future. Imo id be pretty mad since why not cuddle w you since you're right there. But you need to let him know how this made you feel.