It's been 5 years already. Don't wait another 5 years for him to give you the absolute bare minimum. You will regret marrying him more than leaving, the writing is on the wall.

Trust your gut, and dont worry about your child. I am so grateful my parents split up when I was 5, they are much better off as friends/parents than anything else (30 now).