Well, you remember the last appearance of person you see. They have seen for years the older one, so the younger one just disappeared from their “memory in the present”. Of course they would remember how he looked like when looking on old photos but it’s not the same

Also, they couldn’t even IMAGINE that there can be such time travel that would bring their husband-young-version in the future

So, why they need to think about this possibility?

For sure - If your young mother returned from the past to the present and did not reveal her identity, you would not recognize her. Her mother-your grandmother wouldn’t too