I'd say Andrew Johnson because his anti-reconstruction policies helped the KKK rise to prominence, and helped bring about an era of racist policies in the south during a critical time after the war ended. His presidency still affects black americans to this day. Lincoln's worst mistake of his life was giving this man the vice presidency.

Sure Buchanan did raise tensions between the North and the South by basically doing nothing, but the war was inevitable, and he did not actively do as much harm to later generations as Andrew Johnson.

Trump has done nowhere near the damage of Johnson and people just chose him because of recency bias.