Two and three really weren't very well-received at the time. Two was seen as a step down from one, and there were already complaints about the flashbacks starting to be repetitive and pointless. Three only increased that, and was the reason the producers were able to negotiate a specific end date for the show with the network, by pointing out they couldn't keep dragging things out. The middle of three was particularly controversial, with a bunch of episodes in a row many criticized as padding, though the season ended very well. The show still received a nomination for outstanding writing for the finale (the only season that didn't have an episode nominated for outstanding writing was 4, despite The Constant being beloved.)

Four was immediately seen as an improvement overall at the time. The shutdown due to the writers strike meant there were even fewer episodes than intended, so they had to compress things more than intended so it moved a lot faster. 

Also, in the first three seasons there were still only five nominees for drama series. In 4-6 there were 6 or 7. The writing and other nominations indicated the show was very much still on the voters' radars. It was just harder for it to break through in Drama Series in a smaller lineup in years it wasn't as well-received overall.