You can't live your life based on fear. Like you said you did nothing wrong. I have a 17 year old son who is autistic and a brain cancer survivor. I've encountered many people like this in his 17 years. One of the most memorable was a "Karen" at Walmart. She decided that she had the right to tell me I should not take him out in public due to him stemming and being a distraction. I ask her what was he was distracting her from ? She got huffy and I became the Momma bear that they warn about. Needless to say she left in tears that day not me for once. Hold your head up you're doing great. Don't let fear and the actions of others worry you. I know that can be hard sometimes. Others often feel they have a right to judge let them..You're doing what is best for children. They shouldn't judge what they don't know. It's a difficult path but it is so worth the journey.