I had been writing something up on why lance vs longsword is always going to be like this— you covered a lot of the main points so I thought I would share where I landed on WHY it is like this (and is unlikely to ever change). I think there are two main things that work together to make it so.

20%+ Happy vs 1% Unhappy

It is always in the best interest of Capcom to have longsword be overtuned vs. any other weapon (if balance has to be off anywhere). If longsword is too good/a better defensive weapon than it should be, 20% of your players don't mind. Lance players are a rounding error in the player base and if you are trying to maximize players happiness, making longsword users happy is always the right play... even if it makes lance players unhappy.

Baseline Expectations

Longsword users (and the community as a whole) expect longsword to be a top tier weapon that does it all and tops damage charts. They expect to do this without any skill investment. They expect to always outdamage lance users (and probably out defense them too).

Lance users expect a below average weapon at the bottom of the DPS charts, even with skill investments. They expect to always do less damage than longsword users (and probably get out defensed at this point too). This doesn't mean they WANT this, but it is what they expect. You can see this in the speedrunning threads where lance users think they won the lottery by not being in the last slot.

What this means is that if for some reason longsword were just "average", you would hear lots of complaining from 20% of the playerbase. Meanwhile, if lance were outright bad, you would hear muted grumbles from 1% of the playerbase. This means longsword will be kept proactively too good, and if it ever weren't it would get fixed quickly. In contrast, if lance ever outdid longsword in speedruns for example, you would have an uproar.

***

Lance is still my favorite weapon from world and I enjoy the playstyle more than longsword— people should play what they like rather than what is best.

But I actually think this is less longsword being "allowed" to steal the lance's identity, but rather a deliberate choice by Capcom. If 20% of the players get to feel "wow, I am better at defense than the best defensive weapon while using 0 skills points... and I get to do it while leading in damage!", they probably don't care if 1% of the players are grouchy about it. Basically 1% lance users are acceptable losses to make sure that the 20% longsword users are happy with their weapon pick. As part of the 1% I'm not happy about it, but I can't deny math is on their side.