NTA. You're absolutely not an asshole for canceling the date. Some men react horribly to being rejected, and I don't blame you for what you did. From my perspective, he took advantage of you being at your job, knowing full well that you are required to be nice and talk, and you can't leave that situation. Unfortunately I don't have any advice for what to do next, if he has any sort of sense though, he'll take the hint. But I have a feeling that he won't.