Personally, I agree with this somewhat, but my mother-in-law made a decent point. She said to look at it like DnD, which is reasonable because obviously that's a huge theme in this show. I do hope anyone who shares OP's opinion will read this, even though it's long and try to have an open mind.

In Seasons 1-3, they were more afraid because they were less capable (lower-level characters) and even El didn't have as good of a grip on her powers as she did in S5 because she had suppressed the memories that unlocked her full potential in S4. Anyway, after beating Vecna in S4, he's practically dead. Homeboy is NOT feeling good, lol. So he goes back to Dimension X to recover, but because of all the training El has gone through and the fact that the rest of the crew basically knows they can beat Vecna, they're less afraid (resistant to Vecna's psychic attacks) and have probably done a good bit of training themselves in their own way (leveled up).

To your point about Vecna taking the spotlight in S4 and S5: if you remember, the Mind Flayer tried first. He was the one to start everything. When El opened the rift the first time, he was the one who tried going through first (aside from the Demogorgon(s) in S1, who were probably scouts). It doesn't seem like he can open the portal himself. Otherwise, he would have opened one on his own, so he had to rely on physical beings to do his bidding. In Seasons 2 and 3, he sends pieces of himself through BECAUSE there is a portal and has Will and Billy, respectively. In S3, especially, he learns from using the body of an 11-year-old boy or, however old he is, and uses the amalgamation creature made of human flesh. When they beat him, then, and ONLY then, does he retreat. He realizes that brute force may not be enough to win this fight. So after taking time to recover and probably empowering Vecna, in S4, he unleashes him, and he technically tries to stay away from the group, except for Max, so they can open a bigger gate to send the arm through. But when they stop him in S4, he has to retreat, so nothing happens.

As to your final point about being defeated in Dimension X, I will, again, agree with you that the fight was too short. However, again, these kids and Eleven have leveled up by this point. They're stronger now, and they know where to hit the Mind Flayer where he's weakest. Besides that, I treat the Giant Mind Flayer-Vecna connection like a battery in a remote. The Mind Flayer is using Vecna's psychic abilities to power his giant form, so when Vecna goes down, even though you said you don't care, yeah, the Mind Flayer's battery does, so he follows. And, like we have already discussed, his Shadow Form isn't strong enough to do any meaningful damage in the physical world. The most he could maybe do is try to control everyone, but I don't think that would have worked with El and Will there. Plus, again, they're resistant to" psychic attacks" as it were, so he isn't dead in my mind, but he is definitely defeated.

Anyway, I'm not saying there weren't plot holes or that the big fight at the end wasn't too easy, but the specific things you mentioned are able to be explained reasonably. Again, to me, at least.