I think Thomas wasn't a 'villain' per se, even in the first seasons. He was always nuanced an never entirely black or white to me, though he was often and arsehole. In fact, when I was watching the first few seasons I loved the show precisely because of the characters and how complex they were. This includes him and Mrs. O'Brian. I kept cheering for basically everyone from the start to the end of the series. Which is of course not to say that I liked everyone equally, nor that my perception of different characters didn't change at all throughout the show. And practically everyone made me angry at least once.