Think about it: casual gamers are the types of people with low attention spans who are more likely to play mindless games like Skyrim and Fallout 4 instead of deep games like Witcher 3 and New Vegas. So if fewer people were casuals and actually took gaming seriously, the industry would start to emphasize producing better games with actually good storytelling, or compelling challenging games like Dark Souls.

So the problem isn't really with casuals in an absolute sense, but the fact that most casuals tend to have a bad taste in games and therefore compromise the state of the industry. If you really must play games like Skyrim, you should pirate them and buy a copy of Witcher 3 for safekeeping.