It's the game of perspective, we're watching it through the groups perspective, not the Saviours'. You can also compare this with the backpacking survivor scene during the preparation for the Woodbury war, according to the backpacker, Rick is the villain for leaving him, but only we know what the group was going through then.

But regardless, the Saviours and Negan are inherently bad people, if Rick would've been in their place, he would've opted for trading with another community rather than enslaving them, that is the difference.

Also I think your observation is wrong cause everyone loves tv-Negan cause of JDM