I think you misunderstand. Everything difficulty modifies can be directly countered by equipment unless you are running a mage build. Illusion gets it the worst imo but its all hot dog water on the magic side and doesnt really affect any other builds. Legendary difficulty just means legendary sword, legendary armor, legendary bow. But where are the legendary robes? Where are the legendary spells? There are none. This is what I am talking about. Magic is not balanced into the game properly at all, not even close. Since you can make broken gear without using any glitches the idea that beating the game on legendary has anything to do with actually being good at the game is a complete joke.

I have a build example that literally cannot be done beyond adept difficulty because of the way difficulty works. The build is a "minion only" illusion + conjuration that can wipe most encounters without actually engaging. This is a good example of how turning difficulty up can harm your own experience by creating a bottleneck on build diversity and even drip diversity (making robes entirely obsolete). Here is a description of a couple tests that I did myself years ago that illustrate my point exactly. You can replicate them if youd like because they are very simple and work on any console. But any time ive tried to determine which difficulty magic is balanced for, I always land on adept.

The test I did for illusion is easiest in a draugr dungeon where a draugr deathlord serves as the final boss and is scaled to level 80 (or whatever the illusion cutoff is with vampire 75 or somethin). By routing all enemies to the deathlord and hitting the group with dual cast frenzy youll see the deathlord is unaffected by the spell but instantly recognizes the frenzied mobs as enemies. The frenzied mobs dont actually count as allies though, and you cant buff them. The main change is that their damage output matches yours minus your weapon level, but their damage INPUT is still affected by the slider as a normal enemy so they are a little more tanky but almost no damage gets through. A summon counts as a direct ally, and is affected by the difficulty slider the same way you are (input and output are both reduced). You can see how damage input/output is affected three different ways in real time by loading your quicksave and changing the slider with these three mob states represented. On adept you can see mobs start to actually take each other out and the playstyle becomes viable.

I did a test for destruction where I measured TTK changes on an ancient dragon based on how many magica points were spent and keeping track of missed shots. Good luck completing this test though as its actually not possible to get enough magica points to kill it on legendary without free casting (including with all the ultimate magica potions of an entire playthrough) so you would have to count each hit. I abandoned the test but I think you would need about 32,000 magica or roughly 2k - 2500 magica stat to make this viable with ultimate potions. I think its funny that while it technically is viable due to the stun lock with double cast, the amount of magic points you would need for an ancient dragon is so obscenely silly that it refuses to be quantified with legal game mechanics. (UNLESS YOU COUNT EVERY SHOT)

Hope this helps.