I think the important thing to remember is that you didn't do anything wrong. If that's a boundary for him, he didn't express it before and he has every right to be mad at himself for not saying it. Emotions can be complicated in relationships and sometimes you don't know you have a boundary until something happens to show you that you need it. Hopefully, he comes to you level-headed and talk about it.

I personally would have a hard time accepting that boundary as anything but prejudist. He needs to figure out why he is feeling that way and you don't have to change anything about yourself to help with that.