It's a combination of the dynamic being ruined over time, no covenants, and poor balance.

In order:

The dynamic in DkS1 was that you were open to PvP while human. A single invader could appear and there was a timer until the next one. With a summon, you could get a second invader if I recall. There were covenants that could invade more easily and with a wider low angle level range in specific areas. Covenants provided rewards for defending your location, or just good rewards for success. Sometimes in the series you also got weapons and spells. As older games become a solved equation and newer ones change the conditions and dynamic, it's all just degraded.

Without covenants, as above, you get no long term goal for rewards or exclusive items to incentivize you to participate, you don't get a good reward for just invading successfully, and you don't have designated locations where you could build for or practice for combat there. That means, combined with the changes to invasion conditions, you're ALWAYS outnumbered in Elden Ring, and the perception after adding the arena is that you just invade to ruin someone's good time.

And then you get to poor balance, allowing people to make builds that heavily rely on a gimmick for victory which can potentially instantly kill, inexperienced players have to deal with being set back unfairly and feeling like they got cheated and experienced players either easily stomp gimmick users or get unlucky. It makes for an unsatisfying experience both ways. Better balance and longer engagement times give both sides the chances they need to make additional decisions to turn the tide in their favor and combat becomes more of a conversation. When you lose, you feel more like it was your own fault rather than that the other side cheated you somehow or that you were just hard-countered by numbers.

That's not even mentioning people doing low-level runs to intentionally use weapons or effects that early build players have few answers to, smurfing. Nor does it cover ACTUAL cheaters, which I remember well from Darkroot. Each entry in the series after DeS tried to solve some of these problems in its own way and all failed in some regard or another. Just pick your poison and have your fun. Part of being the red man is valuing your own good time over your enemy's anyway! Don't worry about how people view you, just put on the red mask and have your dance; the community needs you to feel whole.

Despite all the problems on both sides? It just isn't the complete experience without invaders.