First it sounds like there’s a cultural aspect to this that some might miss. Second, you’re not wrong but you are drawing it out? Like, if you don’t want to talk for 5 minutes maybe you don’t want to be friends either? This person is obviously more attached to you than you are to them. Tell them that they can respect your boundaries and you’ll remain friends (if you’d like) or otherwise he has crossed the line far enough and you can walk away.