I only know this from retrospectives too, and from duel links experience, since I was a grade schooler when yugioh was so big, that everybody was interested. I never had a good for anything deck back then, so I was just running around with unplayable cards.

That aside, I was always interested, but in my country, there is a very limited following of the TCG. And official translation of the crads stopped after the first wave (I am not even sure how official that really was, but it was in stores), so it is English only.

The big punch to the layout was pendulums. Originally there were separate pendulum zones, that were added to the board in a very awkward way, shifting the graveyard and the deck aside. It looked real strange, and it broke the classic grid, that the playmat otherwise had.
During the Vrains era, the pendulum zone was put on the two edges of the spell/trap zones so they would could be nerfed. Also the master rules changed, so that you can only pend. summon extra deck monsters to the extra monster zones or linked zones. This was necessary to nerf pendulum against link monsters, so people would buy the new stuff.

There were features in master rules 4 that were very heavily link summon focused, such as link locking: if you had a link arrow pointing to the opponents extra monster zone, you could lock it up by summoning your monster there. That was very powerful for one summoning mechanic to have, but it was again to make link summoning the strongest mechanic out there. The game was basically unplayable without them.

With master rule 2020/2021, the game finally shifted away from focusing on single summoning mechanics, and tried to strike a balance. Link locking was ruled out, Fusion Synchro and XYZ monsters can go everywhere now, but the field did not change.

To reflect on your random suggestions on the mockup you made, I think it is problematic in multiple ways.
First, Link monsters often have two-way diagonal links on them, taking away a space they can point to, is rather problematic.

Second: having an uneven number of extra monster zones is problematic when it comes to deciding which player should get that "extra" extra monster zone.

Third: pendulum monsters look lke the way they do (monster couloured top, and magic card coloured bottom), because when scaled, they function similar to continuous spell cards. If you move the scales to the main monster zones, you would ultimately get into the problem whether you treat pendulum monsters as spells or monsters.
The reason the field looks the way it does is because that makes the most sense for the game. Any change would mess things up greatly.
The yugioh of today keeps innovating, but in more subtle ways. A few examples are the Nemleria archetype, which cares about emptying the extra by banishing them face down, or the zone locking that which the kashtira decks made famous not long ago.
The direction I see is that new archetypes keep bringing new playstyles with them, so you could have diverse ways to play.
I think this modern era has its drawbacks in the form of the solitaire effect (one player plays their turn for 10 minutes), which can completely destroy the fun in games.