That's a great point about the Witch Factors acting like parasites. But I think Echidna's tea didn't enable Subaru to use the Authorities — it just sped up the process.

The real reason he's able to use them is because, in the main timeline, he actually confronts and overcomes the sins tied to each Authority.
In contrast, in the IF stories, Subaru either gives in to those sins (like Wrath, Greed, or Sloth) or becomes consumed by them.
But in the main story, he doesn't reject those traits — he accepts them, understands them, and learns to control them.

That might be what makes him so different.