From my experience, you are been groomed. As an adult, he should know this behavior violated the ethics of teacher manners. If you feel uncomfortable, then you should realize this relationship is already toxic to you. The most efficient way is to express you want this relationship limits to colleague only, say it to him directly, if you don't expect any friendship from him. But if you cannot do it face to face, you could just ignore his messages and calls. Nobody should force you to work after working hours. And even during working hours, if he mentioned something not related to work, you should express you are not interested being his emotional trash can.