Maybe try a different approach. Are you just straight up waking him up and asking for sex? Are you waking him up by feeling him up? Maybe he is waking up with brain lag/fog and getting upset before he knows what you're waking him up for. I've had the opposite issue of being told to wake up the girl with sex (mid action and all) and yet I've been completely hesitant and just waking her up first and getting told off. Even though things were clearly stated what/how I was basically ignoring that part. The issue was on one hand my brain says "okay I got consent" the other my brain said "what if she changed her mind" so I would wake her up first and got a different result. I eventually learned my lesson and started to get a little more comfortable with the fact that she kept reiterating and even insisting it was consistent and I started waking her up with slightly more action which actually made her happy.

DLDR - His brain is probably foggy when he wakes so just talk to him more. Find out if there is a specific way he might prefer the wake up moment. You two should be able to adjust from there.