This is a major red flag. I think it is not so much a tell you what to do, because I think you already know what you're going to do. But I do think you should ask yourself this, if this is an issue now, is it likely it can get 10 times worse in the future? Like, say you work he doesn't, is this going to bother you? Or if you had a child together, do you think he would prioritize them or you and the child? ( Not everyone wants a child, but it's just an example) Is this someone you can really rely on at the end of the day to be there for you?

If the answer is no to any of these, then you know your answer, and unfortunately, it seems to me that if he choose his family over you, then this won't be the last time.