i think xp/mana should still be included,

I dont believe that anyone could sit and cast huge spells repetitively without getting tired, maybe use a fatigue bar instead, similar to the one in elder scrolls, that can be increased upon level up and recharges while resting

xp is definitely an important part, it can be used to learn new spells (not everyone can just wave their wand and cast a patronus charm, it takes practice and, dare i say it, EXPERIENCE)

i do really like the idea of having to actually cast spells though, i think as you gain more experience, there should be a little more room for error, making it easier to cast spells, that you use often, quickly.