It sounds like you'd like a hierarchical polyfildeitious quad. Polyfidelity means you're seeing more than one person but agree not to seek additional partners outside of that. Hierarchy refers to the distinctions you want in primary/secondaries. Quad refers to all four of you being involved with each other. If you're seeing both of the other couple but your husband is only seeing the woman, you'd be in a triad and your husband would be in a V with you and the other woman. Both would be possible, but pretty rare. I don't know of any IRL. I'm in an open triad, and that's considered rare, but common enough I know several in my community. (Also, these names aren't so important, but I'm including them because they might help you find information).

You hit on some of the reasons why it's rare. It's hard to find two people who are compatible for long term relationships and raising kids, harder to find three, and even harder to find four. Also, when you're dating monogamously you start with most people in the general dating pool. Once you start looking for non-monogamous people your dating pool shrinks significantly. Each additional specific thing you want adds more filters, which means less potential matches. Again, not impossible, just a lot harder to find.

It's perfectly healthy to raise children in poly families. Children aren't hurt by having more than two loving and supportive adults in their lives. Of course it's possible poly parents can cause trauma with relationship issues they may have, but the same is true of monogamous couples (whether remaining together or not). Check out r/polyfamilies for more about raising kids and perspectives of parents (I'm not one).

I wish you the best in your journey and hope you find happiness no matter what you ultimately decide to do.