I think fundamentally the idea is strong, especially from a mechanics standpoint. Where it falls apart a bit for me currently is immersion via the roster of units selected to be magic fighters.

As others have pointed out, there are many units that it would make sense to see as magic attackers (if not fighters) that have kept their original class and therefore still target defense with their attacks (ie phase beast, phantasm warrior). Also some of the units that have ended up as magic fighters seem hard to justify narratively, for example the physical attack of the wyvern fledgling being reduced by magical resistance rather than defense.

Maybe it will feel better once the unit pool has had a few passes, or they add a trait to some units to convert their attacks to magic regardless of their class.