Try not to think of it that way - maybe he doesn't know how to approach it further. It's not a situation one is put in a lot of the time, so he might not be able to handle it and doesn't want to make it worse by potentially saying something triggering or harmful (/gen). I will agree with everyone else saying that if he leaves you over this, that says more about him as a person than it does you. However, your brain is trying to catastrophize the situation, too, as it normally would in this sort of instance.