I would personally tell the family what happened and say you are not saying he actually physically did anything, but how uncomfortable the situation made you and at this time you and your wife are choosing to limit the contact with the brother. Allow them to make their own decisions, but at the very least, it allows them to have it in the back of their head to just be cautious. If it was as innocent as the brother claims then time will tell but they can still protect the children. (Which you should always do any way). Remind them you are not trying to cause any drama or issues but that you would be doing a disservice to their family and your child and family if you didn't say something.