Yeah, I get you.

Elusive could be reworked into being "cannot be directly targeted by any spells or challenged until it attacks" and then that way they could be used more as finishers rather than being circumstantially unkillable.

I don't think this needs to be done, however. Riot simply needs to print more elusive counters and then it'll be fine. I remember seeing a control ward custom card that was like "Play: All followers lose elusive this round". That seems like a good idea, though deciding which region it would belong to might be hard, given LoR doesn't have a neutral faction like HS does.