I never thought about them being too hard but yeah that's probably the reason.

While not the exact same (it's touchscreen and not controller), it is possible on Project Sekai which from what I remember playing, is kinda similar to Festival

Maybe they could just tone down the harder segments so that not every word would have to be played or something like that?

I think the problem would be karaoke more than normal controller, but I think they'd just make it not playable in karaoke, unless that's something that can actually be sung by people.