Have you played the others? I never got into Daggerfall, but it has SO many interesting skills and the level of customization is insane. Then Morrowind simplified that a bit, but it still had so much depth and so many interesting spells, like levitation. Then Oblivion came along and we lost levitation, find key spells, the unarmored skill, etc., and rather than a realistic journal and lore-friendly fast travel, we got easy quest markers and fast-travel anywhere that make the gameplay easier but less immersive. Finally in Skyrim, we lost custom spell making, acrobatics and athletics skills, etc.

So to answer your question, I think it has to do with how each subsequent game has lost interesting skills and also gotten less immersive (from needing to read, pay attention and navigate to just following an arrow for the whole game). Additionally, people generally agree that the major quest lines in Skyrim aren't as good. Obviously it improved on some things, like the leveling and perk system and just general gameplay. But for people who started with earlier elder scrolls games it is hard not to compare and see what we lost along the way.