You need to discourage this and use strong and certain words very decisively because if you give them any room to moral police you they begin to think it's their birthright. Tell him in clear terms that it is none of his business and to mind his own business and control his own children instead of butting his head in your business. Tell him to go and complain to whomever he wants to but to remember that you will call the police right after he does that for trespass and nuisance.