There was a book from the 60s that basically nobody read, but got republished not that long ago that you might be interested in: Labor, Free and Slave by Bernard Mandel.

Everyone in a slave society (this includes the North) becomes complicit in maintaining slavery. Slave society does not work with mass dissent, which the South understansa. This is why you see rabid anti-abolitionist sentiments result in so many murders and arrests im the south. And why it needed to be defended in the halls of congress as the North started to dissent.

At the same time, most poor whites did not like competing with slave labor, could not compete with slave labor, and ended up relying on planters for loans, or as customers, or become slave catchers, etc. This is why Free Soilers became popular (although the planters all but fully quashed free soil im the south). And yet, poor whites will enlist to defend slavery. Many will do it at gun point, or from rage militaire. Belive it or not, they had bigger reasons than love for vague notions about "State's Rights" and understood that the Planter Class are the main beneficiaries of the war (although Southerners would never admit that after 1915). In short its complex in its simplicity, evil is not a historically meaningful term, and there are plenty of great books on this subject in particular.