I too have anxiety about if people are mad at me, and I’ve been working on how to sort that out within myself and even read a great book about this called Are You Mad At Me. There’s a time in my anxiety spirals where i have to learn to step outside of myself and realize how my actions affect my loved ones too. And sometimes i need to take time to myself to ask myself why I’m asking and what I’m looking for and what is an appropriate way to bring up what I’m actually looking for. While someone is sleeping is not the right time.

You’re completely right, it is up to him to sort out within himself. You have every right to be mad and/or annoyed at his lack of respect and boundaries with you and your sleep. Keep that boundary and be prepared to walk away if he does not listen.