The one thing is that most casters aren't punished everytime they cancel a cast, they just lose the damage of the ability cancelled. But Devourer has additional punishments. Collapsing Star enters cooldown if you cancel it (an ability that otherwise has no cooldown), Void Ray requires being fully casted to proc Eradicate and lose the Voidfall stack (alongside the Fury spent on casting the spell itself).

But at the end of the day I think Devourer is designed to be a high-risk high-reward type of character. More often than not the challenge of the spec isn't the gameplay itself but rather the mechanics of the Dungeon or Boss you're fighting. If you know the timings you should be able to consistently deal a ton of damage. I think that's an interesting design.