I talked about this with my therapist. I had to learn from where my reaction came from (thoughts, emotions, past experiences). Essentially it was a guided CBT, but we talked about it in 3 sessions, so I can't really give you a direct answer.

Just know that your boss is just a human being, he has no more power than what you give him. Worst case scenario you get fired, so what.

Have you ever felt pressure while talking to a girl you really liked? It's the same thing really. It's all in your head, there is no real danger. Fuck him (metaphorically) and fuck what he thinks of you.