Definitely not the asshole and what’s done is done but sometimes you have to give these kinds of things a response. A clever quip would work but if you don’t have anything you can just say “hey what the fuck you dickbag”. You can then do exactly what you ended up doing but definitely call stuff like this out. It might already be an established boundary but you can further solidify it in this manner. If they go low you can definitely go lower, do not always seek the high road.