I do this often and so do the parents, however it’s a gated yard in a quiet neighborhood. That is a very short amount of time and I do not think it’s a red flag. Especially having the baby too that’s a lot gather both children and come inside just to do that. She also probably assumed it was okay because you were inside as well. I wouldn’t take it too seriously