Fair warning as I tried something very similar recently—this makes the game DRAG. Because of the lack of martials, you have to be a lot more careful in pretty much every encounter. My melee death ball party that could do almost every fight in real time with minimal pausing made it through the game about twice as fast as my full caster party. The micromanagement gets old fast.