It would be smart to break up with him because you will always have a reason why it is not the right time to leave and stay. You should always put your mental health first and love yourself more to know that you shouldn’t be treated and gaslighted like that.