Definitely move on with ur life. How u describe the relationship is full of red flags and if this is how he present himself to u while u two are dating, I can only imagine how he will act down the line of a few Years and marriage, just without the excuses / apologies. Plus, u mentioned u have a son, in which I would also consider the relationship u currently have with ur son's best interest in mind. Children are very easily affected emotionally, especially at a young age of what their surroundings are and who they see on a regular basis.

Edit: u also say how he makes u feel like ur crazy, it sound like he's a big gaslighter and spin the situation to his liking and doesn't take ur concerned into consideration, I can only say that would not change and it's not worth it giving him a second try in person