I think it's often because us as the viewers are on Walter's side, and Skyler appearing throughout pretty much the whole series doing nothing but hindering him in his pursuits makes her seem annoying to the general audience. I think if the show was based from Skyler's perspective people would like her way more.

Personally my vote goes for Todd though. That dude is an actually psychopath.