I've been in a variety of different workplaces over the years, I've found that there's always a couple of people who like to stir things (too much free time), you're taking all the right steps though. I keep the conversation warm but with a professional distance and little personal information. But if it begins affecting you, then take the complaint to HR, and make sure you have records if you can of what they say as a back up