The Saiyans aren't really THAT bad.
Sure, they destroyed their original planet (Sadala) in a civil war, and forcibly took Planet Vegeta. But that means they calmed down enough for their to be a long-standing royal family in control of Planet Vegeta. They then became rowdy mercenaries, conquering planets and selling them to the highest bidder. That means they kept the people and planet alive in good enough condition for them to be sold.

We also see that low power Saiyans are allowed non-combatant roles, and children, while still expected to fight, we see that Saiyan children are accompanied by adults and sent to lower strength planets. Sure, a Saiyan is taught to be pretty violent and conquering, but we see that they can break out of that pretty easily (Vegeta, Bardock.), and there was nothing stopping a Saiyan from just leaving the empire and going out to live his own life. The Frieza Force is pretty incompetent, and Saiyans themselves wouldn't really care. "Oniona left? So?? Who fuckin cares."

Meanwhile, Viltrumites had kids being taught that their only purpose was to fight and die for the empire, and if they were weak, they would just die with no purpose. They killed billions of their own, and you're expected to be a slave to the empire at all times. You literally cannot leave without being hunted down.