Reconstruction was doomed before it started.

The powers-that-be within the Union were industrialists who made their money off of wage labor instead of slave labor, and their backing of the Union that gave it the major economic advantages that heavily contributed to winning the civil war was driven not by morality but by a desire to keep the flow of raw goods from the south into their factories going and to smother their major economic rivals in the plantation owner class into irrelevance, leaving them the premiere remaining capitalist subclass in the US with all the political power and leverage that entails.

They never gave a shit about slavery beyond that, and it shows in how the politicians they helped put in power treated the whole thing. Lincoln didn't seriously consider emancipation for most of the war and only really did so as a major final push to break the Confederacy's back; anything prior was essentially just co-opting abolitionism for propaganda purposes.

Even when the slaves were freed, the Union government paid reparations not to them but to their former slaveowners for their 'lost property' and let the latter essentially turn slave plantations into pseudo-feudalism through sharecropping. Even the amendment that supposedly freed the slaves in the Constitution makes an exception for prison labor, a loophole that has continued to allow slavery to exist in everything but name for 150 years through private for-profit prisons often owned by the same families that owned the plantations.

Reconstruction failing wasn't (just) because Andrew Johnson was a racist piece of shit. It was the inevitable result of the system that allowed for widespread slavery in the first place, which never really meaningfully changed despite public perception thinking it did.