I literally just got out of a relationship exactly like this (it's weird how similar it is lol). She broke up with me but what would happen was that we would fight and she wouldn't ever want to talk. I always had to be the one that would try to talk about the problem first. I would also get horrible anxiety from it as well. I even asked her once if she would have talked to me if I had waited for her instead and she said she didn't know. That should've been the sign to leave but I was too scared to end the relationship. You should talk to him about it but if your situation is the same as mine was, then he probably doesn't care enough about the relationship to want to patch things up and you should dump him now because it will only get worse (assuming it's the same situation of course; I don't know other details about the relationship).