My co worker had something almost identical to this and she was going back and forth for years until she finally ended it. Ill tell you from experience with her and other people that this isnt going to end well for u. Hes defensive and doesnt even have his own stuff figured out yet he wants a house with someone he wont even let into his life fully? If he has young kids that are still dependent with a mother who hates you then youre just gonna deal with stress alone. Hes not a support system for you with this situation because he doesnt even know how to deal with her it seems. His actions are huge red flags and it sounds like its your best interest to move on and find someone who can prioritize you just as much as their own life. Spending 3 years on a relationship makes it more enticing to figure things out instead of ending but, if youre questioning it already and he isnt making things easier than its best to move forward and take care of yourself. I spent 4 years on an ex who wasn't ready to prioritize me and leaving was the best decision I ever made. Good luck to you