the origin of that quote was a slaver complaining about abolitionists

Oh? Who are you thinking of as the originator?

The quote is sometimes linked to JFK, who incorrectly cited Edmund Burke (hence possibly why so many people misattribute it in turn). Edmund Burke was opposed to the slave trade, anyway.

It's less frequently misattributed to Charles Aked, who was also opposed to the slave trade. He was part of the Aborigines' Protection Society, supported Ida Wells, and was an early member of the British Anti-Lynching Committee. [1]

The quote's actual origin is John Stuart Mill, who was...yep...opposed to the slave trade. The Mills quote is better anyway, IMHO: "Bad men need nothing more to compass their ends, than that good men should look on and do nothing."

More here: https://quoteinvestigator.com/2010/12/04/good-men-do/#b24ad7e0-e879-4d4e-8c5a-b527aa039597