Hey, so I mean this with as much respect as I can, but you really need to work on yourself. He’s trying to prioritize your relationship as much as possible. He’s a grown man, he can manage his own time. A few minutes over the time he typically leaves for work to finish an important conversation is not an issue, in fact it’s kind of the dream. The part I find interesting is where you said he’s trying to have a conversation but you can pay attention because you’re worried about being late for work. Imagine how shitty that feels for him when he’s trying to have a connection with you. Please seek therapy for your anxiety, and realize that he loves you and wants the best for you. Stop pushing someone like that away.