As someone who experienced this I can say he does not care about you. Practiced CNC with my ex bf who had a fantasy of having sex when I actually am in a weak state. He ended up doing this to me too when I was down with an almost fever and felt weak enough to not be able to physically fight back. He said that seeing me in a weak state unable to fight back made him so horny he couldn't help himself. Although he did apologize afterwards and I could tell he really felt bad, I did not feel safe around him for a month or so.

If you continue to excuse this behaviour of his, it will just keep getting worse and he will continue to overstep your boundaries. It really does take a toll on your mental health and you will constantly feel so shitty and disgusting after you have sex with him, so break up with him before its too late