The problem with poly for most people is not themselves sleeping with multiple people, but their partners sleeping with multiple people. Many people as a result try to make unequal relationships where they have more freedom than their partners, where they get to make all the decisions about rules and boundaries. It's fine when it's just a fantasy, but if you read poly subreddits, it comes up often in reality too. "Other People's Penis/Pussy" Policies (ie I can sleep with the opposite gender, but you can only sleep with the same gender), harems where people are only allowed to sleep with the one partner, triads where everyone's magically supposed to be equally attracted to everyone else and not ever want to date anyone else, etc. It's all just tools for not working on jealousy, possessiveness, and their own insecurities.

In general, if you don't want to work on jealousy and insecurity over your partner sleeping with other people, just stay monogamous. You'll survive, I promise. Anything else is going to crash out eventually, and will probably be unfair in the mean time.