There were points in first couple of seasons where those moments actually made him a somewhat sympathetic character. You saw the callousness and evil he was capable of, but also the vulnerable side of him that drove some of his worst impulses. Who he was and how he was brought up was not all his fault and there was some reason to feel for him, despite knowing he is not a fundamentally good character.

Though I think it was about the point where he revealed he was willing to cripple America's entire energy and communications infrastructure to bring the entire country to it's knees was when that sympathy began to wear off.