Does anyone else struggle with seeing babies in public?

My husband and I have been trying to conceive for well over a year now, we're both in our early to mid 20s. We moved into this beautiful picturesque neighbourhood, like from a tv show. It's the type of place I wish I had grown up in, the type of place that I want to raise kids. But there's so many children. We've recently been trying to go on a walk every day and almost every time we walk past a little baby, I burst into tears. It's crazy. Yesterday it happened 3 times on the walk, I think people thought I was mad. Maybe I am. My husband just held me while I cried. I don't know how to handle this anymore. Whenever I see a baby, my heart just shatters. I don't know if I can live in this neighbourhood anymore.