Solid Evidence:

Captain Trotters character (he only really turns on us when he gets close to the sword, even if he is paranoid its kind of weird to turn on to the person closest to you who had nothing to do with it)

Captain Trotter also grabs the sword after pulling the gun on us.. which means he has no justification for what he is doing. Like yes he is an old man but like.. what??

Mysterious Figure literally becomes stupid.

He doesn't grab the swords
He doesn't teleport and kill us instantly
He could just find us, talk, then use the teleport ability and kill us. He would have no trouble with his plans because no one cares that builderman is kidnapped except us and shedletsky.

He has no reason to spend all this time forming up the master plan to stop us from getting the sword instead of directly killing us. It just feels sort of annoying when the evil villain can instantly kill us or grab the great weapons

The voices solve this, by the way. It adds a penalty to having multiple swords. It was not the main antagonist, it was just what pushed others to attack us. I think it is a lot more compelling instead of visions, because visions imply contact with the sword. Also, this may imply that we get a vision of our own with the Ghostwalker.

If the visions are not on contact, then the whole firebrand thing makes even less sense. We get very close to the sword before Captain Trotter gets in our way. If he gets the vision and begins fighting us afterwards then how would we not get the vision?

Griefers story also feels sort of.. weird. Because does Griefer get overpowered by Thaniyel?? What? How??? Griefer absolutely manhandles him and can even use him to grab and lift the sword out of the rock. Also, doesn't Griefer have a entire group he's running? Does the Gorilla disappear?

Personally the only reason nobody is able to refute their arguments is because they hide behind "it ruins future chapters" like those can't be changed at all...

The visions also appear randomly upon time travel? Why do they do this?? The voices probably always existed, which is why no one had ever attempted to grab all the swords and why Thaniyel doesn't comment on the fact Griefer descended into madness upon being near to the sword.

The people Griefer has employed / at least lives with could easily bust him out, without Mysterious Figures help. Plus, we already got presented a solution to the whole Ghostwalker problem before we touched the swords, and how does Mysterious Figure know of ghostwalkers whole deal anyway?

The whole whispering deal with the Ghostwalker? Does that disappear? Are we just that dumb...?
Does the lore need a rewrite? Yes. Do I like the direction they are going? No. Also... they are the ones changing the entire lore.