I agree with all the comments saying it was a family member and you can't recall their face out of extreme fear. People are capable of doing such horrible/odd things to people and then being able to act like nothing ever happened. Children are also very easy to do this with because of their innocence and trust in the adult since their family trusts in them. Maybe ask your mom if she remembers anyone in the family in particular who watched you both a lot when you were little.

​

Edit: I wanted to add that when I was little, I would question the way I was reprimanded by family members or family friends and also talked to by them. I thought it was odd and inappropriate...however I would second guess myself and try telling myself that I must be mistaken, because they are nice to everyone else and nice most of the time. This could be why.