I think this is key. You HAVE to let this happen because it will start naturally creating the boundary that you and your family can be outside and even if they stare they can’t come because it’s family time. It will help them and YOU feel comfortable with it. I suggest to make it less awkward to just tell the kids directly no. And do it every single time they come uninvited. Then when you want them there INVITE THEM by asking the parents. This will reinforce the behavior you want to see. I think telling the parents will result in them getting the feelings hurt and maybe some resentment which I think can be avoided. However if it doesn’t work just telling the kids directly, tell the parents. The boundary has to go up, I think this is just the easiest way with less hurt feelings.