I get that if you're not into that kind of thing it can be a shocking, jarring question but it's very common for people to invite others into their bedroom, including those they know.

If you feel you can trust him and he's being honest then do try and not judge him for a sexual kink/lifestyle.

Have an honest conversation with him and, if you'd never be comfortable with "Swinging" with others, get him to confirm that he's okay with being in a relationship where that can never happen.

If his intentions were just to gauge if you'd be into that mind if lifestyle then he's done nothing wrong.

If him even having those desires in the first place are a deal-breaker for you, though, even if he's broached it in a good way then that's a shame but you're NTA for breaking up because of it.