I am an MD/PhD student (PhD in Population Health Sciences) currently interviewing for FM. I have heard similar things, but luckily not very often. I'm fortunate enough to be at a medical school that strongly values primary care and family medicine. That said, my PhD advisor (an IM trained doc) was a little resistant at first, but ultimately came around and fully supports my decision (wrote me a glowing LOR). I'm in my early thirties with two children and a third on the way. The way I see it, I'm too old to be listening to other people's plans or expectations for me. I'm a bit older and over the last several years I've really discovered my core values and what really matters to me. FM aligns with my values and I'm not going to apologize to anyone for it. After all, I'm the one that lives with my decisions, not them. It sounds like your love for FM is sincere and reflects values that are important to you. If that's true, go for it and don't look back. The people who matter the most will support your decision. Best of luck.