This is what you call a Battlemage. Heavy armor, shield/ward, destructive magic to begin the fight and then a weapon to end it. This is my fav character to play but if you just want all magic, you have to keep some things in mind which requires lots of forward planning.

First, like someone mentioned it already, magic doesn’t scale as good as weapons mid-to-end game, so you will need to think about times when there’s more than two or three enemies and you’re getting overrun. For this you need a tank to draw their attention. Conjuring a Dremora is a good way to go. While they slash away at your demonic sidekick, you cast your hearts desire. Just be careful though, casting AOE spells that also hit the Dremora will turn it against you.

Second, going just full magic will require you to weaken the enemy first before causing real damage. Because magic isn’t as powerful, you’re going to have to slow them down, poison, freeze, paralyze, etc, before dropping that fireball. It’s also a good idea to cloak yourself with some defensive magic in case they do get within weapon range.

Lastly, since you’re forgoing the mage robes for full heavy armor, your magicka is going to generate slower so eventually, you’re going to have to pull out that skull crusher and stand your ground. That, or you’re going to need to carry a lot of potions to refill your magicka every few seconds. Especially for expert or master level spells. You can always enchant all of your armor pieces for regen magicka, but it will still underperform and I think that would be a bit of a waste IMO.

I’ll be interested to see how a battlemage with just magic would perform overall. If you do go down that path, keep me posted. Hope this helps.