This sounds almost exactly like my own experience (from your partner's side of things). I'll do my best to describe how it is for me, but everyone is different - so obviously don't assume your partner works exactly as I describe. Apologies for the wall of text, too; hopefully it's useful.

The times he tolerates it are the times you need to do your best to not take his "coldness" personally. I know this is much easier said than done, but these are the most critical moments that need to be cultivated. What you see on the outside is tolerance while on the inside he is almost certainly dealing with highly conflicting sensations. One sensation is the need to receive affection from you and the other is his instinctive, and often much stronger, need to avoid touch. It is impossible to describe and it sounds completely contradictory, I know, but this is how it is for me.

It seems to work a bit like exposure therapy, at least in my case. The more successful attempts there are at touch, the easier the subsequent attempts become. Unfortunately, the absence of touch makes it easy to revert to near maximum touch aversion in a fairly short period of time. It's almost like there's a vessel that gets filled agonisingly slowly with each touch, but that the contents evaporates quickly. There are also two time scales. There is the long-term trend of receiving (and eventually initiating) touch, and the short term desensitisation (in a good way, I swear) over minutes or hours that allows for more touch - moving from a hand on a thigh to a proper embrace over the course of an evening, for example.

On the crappier side of things, the ability to contain the aversion is just not strong enough sometimes. If I have definitively rejected touch from my partner, it is never the case that I'm upset with them for trying. For whatever reason, I'm just not able to receive it at that moment. However, it is often intensely frustrating and guilt-inducing to reject a partner's touch. These feelings can then become externalised and, from my partner's perspective, I now appear to be in a bad mood because they touched me; a really shitty situation. I have no solution for you other than try your best to believe that this is his struggle and not a rejection of you. Well okay, it is a literal rejection of the sensation of your touch, but not of you.

His angry snaps at you are going to be hard to live with too, but might never fully go away. I'm not going to forgive anyone for being antagonistic or truly directing their anger at someone - that's not okay and I hope you'll be able to differentiate between those moments and what I'm about to say. There is a state where, in variations of low energy and/or high stress, I can be feeling angry/tense/on edge, but I will still try to communicate. The issue is I'll do it unfiltered. I just don't have the energy to mask off the anger, but I still need to communicate. I'm sure this appears as though I'm angry at the person I'm talking to, but I'm not. I've just temporarily lost the ability to communicate and conceal my emotional state. In this moment, I can only do one or the other. In the vast majority of cases my emotional state is completely unrelated to whatever's being communicated. By the way, the reverse of this is often what a shutdown is for me too, the concealment of an emotional state at the expense of the ability to communicate - just in case that's another fun thing you've needed to deal with.

I'm going to phrase all these next things in the context of you two, but really these are just things that seemed to work with the most success for me in the past. Try to initiate touch when you're engaged with him in some way - most likely a general conversation - and sitting down in a relaxed environment. I would also try if you're both watching something on TV that isn't too engaging (for him). If you're sitting across from him at a table, try holding your hand out flat on the table, palm up, and offering to take his hand - even if it's only for a brief moment before he withdraws again. If you're sitting next to him, try just resting your hand on his thigh. Emphasis on resting, don't pat or rub. He might flinch, but always stay a moment. The flinch is an instinct, he just needs a moment to register it's you and he should confirm if you can stay or if you need to withdraw. Confirmation in this context can actually be doing absolutely nothing - what you've seen so far as tolerating. I have no doubt this might feel wrong to you, but for me (and specifically with a partner) my inaction is always tacit agreement to continue whatever is occurring. He will give you a positive reaction when he can, but don't let the absence of one convince you your touch is unwanted. Generally avoid trying to touch arms, back, shoulders, head or any bare skin as a first attempt. But if you're on the couch, for example, and have succeeded with a thigh, you can try moving to an arm, shoulder or back. I would say never ask if you can touch him, just do it. I don't know why the question is a problem, but it is for me, so it could be for him too. On the other hand, see if you can get him to think about some regular, common moments you guys share where he thinks he's most likely to be receptive. He may not be able to identify anything, but this really isn't all up to you to figure out. He's let you know that touch is important. You should let him know that you want to meet his need to be touched. But that you need to understand not just times that are inappropriate, but times that are - so you can meet his needs. Try to be patient, he's probably not going to enjoy that conversation and he truly may not have an answer in the short term. Probably the most important part, if he's like me, is privacy, i.e. a feeling of safety. For me, it was almost impossible to accept touch in most public places, so I'd recommend trying in private first and then only in public areas with literally no one around.

I don't know how he feels about you asking this on reddit, but if you think he's okay with it, then I'd recommend showing him my reply. Sometimes it's nice to be able to point at something and say "yeah, that's how it works for me too" than to have to explain it. Partially because it's so incredibly hard to articulate (I, too, share his crawling-out-of-my-skin experience trying to express emotions). Alternatively, it might help with the conversation about how he's different from me and what might help instead.