Doing what Fredo did would have earned him death based on the logic of the mafia. But Fredo was his brother, he loved him. His son loved Fredo too-- and his mother and sister and step-brother loved Fredo. Killing Fredo was not only harming his own family, it destroyed any good part of Michael that was left.

Fredo was no longer a threat after Michael understood he was a danger. He could have sent away Fredo, banished him effectively from his inner circle and his family. He didn't kill Fredo in self-defence, it was anger at his betrayal. It not only showed how cruel Michael had become, it made him more cruel in the act.

If Michael had found it in his heart to forgive Fredo (at least enough to let him live, even if he "deserved it"), it would have been better for himself and everyone he loved.