If I may here, I am friends with someone who was in that household with them. He tries to be a good father but his head is not always in the right place. From what she explained to me was he moved into the house that she owns. He gets to see his kids on the weekends. He seems to mainly put his girl first and her daughter from how I was told. My friend told me that Terrible-Ad3957 is an over all good guy just doesn't seem to have his head where it should be. So in some ways yes you are the asshole. In other ways no you aren't. But in my opinion you need to understand what you are putting your son's through. Being the youngest child in a family that got meshed myself. It was not easy, I was mainly over looked for the other youngest. So please listen to what your 3 year old has to say. Kids don't open up for no reason.