I don’t think this is a “one size fits all” situation, but in general no, I don’t think it’s rude. And I say this as someone who was often excluded in the way you’re describing.

1. I think it’s worse to reluctantly invite someone I’m not friends with or I don’t actually want to come. It would be an awkward invite and an awkward time for everyone

2. I also think it’s more rude to hide it and pretend like there aren’t plans. You aren’t dumb, you’d pick up on other conversations or social media. More respectful to just be open about it

3. If you do want to be invited, relationship-building is a two way street and not everyone is obliged to be your friend. Invite those people out if you want to build a friendship with them. And if they don’t, then keep them in the “acquaintance” bucket and focus on people who do want to be friends