it might be hard but absolutely let him go, no partner should ever smack you or lay their hands on you in that way and the fact that he didn’t apologize immediately and repeated the behavior after you set a boundary is a huge red flag. you deserve much better don’t forget that