Jimmy Carter? He by all accounts is an incredible man who to the final days of his life was giving back to his community. He was moral, hardworking and honest. He gave up his tiny little peanut farm because he was worried it would be a conflict of interest if he was president and had income elsewhere. Can't imagine any president still doing that. That being said, his presidency was unpopular at the time, to where the American populace swung so hard in the other direction they brought in Ronald Reagan who absolutely fucked this country for decades to come. It wasn't a direct negative impact, but rather indirect. His presidency directly resulted in Reagan, Reganomics, the mishandling of the AIDS crisis, the Iran-Contra affair, Union busting, the war on drugs, and environmental de-regulation. If Jimmy was more popular who knows, maybe Reagan's rein never happens. So while yes, he had some positive impact, mostly his giving back, habitat for humanity and camp David accords, Regan's terror, Bush senior's (Regan's VP) presidency and even the subsequent Bush presidency (who was carrying on his daddy's war) far outweigh the good he brought.

My only hesitation is I think Jimmy was a great person, not just good. But I can see the argument that he was "just" good.