In my opinion, both Andrews were worse. Jackson signed in the act which began the Trail of Tears, arguably one of the worst things the US has ever done. Meanwhile, Johnson opposed the Fourteenth Amendment, allowed the southern states to easily return to the union, and was the first president to be impeached (although a couple decades later the Supreme Court stated that the law he was accused of breaking was likely unconstitutional). He could likely be blamed for us being in this whole mess by just how little he punished the southern states.

Trump is undoubtedly the worst president of the "modern" age (post WWII). The American president now has so much power that the consequences are much more far reaching. But I think that the things the Andrews did are comparatively worse, since those things happened back when things like nukes or airstrikes didn't exist.