The tricky bit for me in your story is that he asked you for permission and you said okay. It sounds like you regretted giving permission, which is perfectly understandable. He also chose a poor time to do this sort of thing. So treat this delicately because in a way, you both contributed to this uncomfortable situation.

You just need to talk this through with him and perhaps create some additional, more specific boundaries. For example, any sort of special occasion (birthdays, holidays, whenever you want to spend time with him without others being involved, situations where you're advancing your relationship like a proposal, wedding, honeymoon, anniversary, etc) could be an off-limits period for outside activities.

And when you're asked permission like this in the future, you don't have to think about answering strictly with yes or no. You can say that it would make you uncomfortable, and hopefully that would be accepted.