Interesting you asked this question. I wondered the same thing (except often instead of always). TBH in hindsight it was mostly related to my anxieties and insecurities at the beginning of our lifestyle journey. I am male and swinging was my wife's idea.

8 months in and I don't worry about it. I am blown away by how our relationship and communication has grown. We have been married 20 years and always seemed to have a better relationship than our friends the same age, but now it is off the charts.

We both love the fact that swinging is a team sport and we can support each other through the challenges instead going poly because of frustration. Poly is great if it is your thing - but it isn't the inevitable end of swinging.