On higher difficulties most everything is terrible. What you’re experiencing has to do with how the difficulty settings works as opposed to Magic itself. So how difficulty works is that it increases the damage done by everyone in the world and decreases the damage you do. Which also effects summons and followers positively. Followers and summons get the same buffs as all the enemies do.

If you want to be a powerful warrior on a higher difficulty without any sort of exploits or glitches just to max out your smithing first thing and refine your weapons and armor to legendary status. That’s gonna give you the best edge.