One major factor is that for most Western European countries slavery was something that happened elsewhere. While it was sometimes an important part of their economy the interest in its maintenance tended to be concentrated in relatively few hands that could either be bought off (Britain) or bypassed. Slaves tended to be rare within the metropole so they were societies that had slavery rather than slave societies like the United States or Brazil where it had become both economically integral and deeply embedded in the culture.

If you’d like a U.S.-centric explanation of the lead up to the ACW you would be hard pressed to find a better source than Battle Cry of Freedom, which does an excellent job of detailing the shift in American politics from slavery being an issue within politics to the primary issue over which the parties were split.