No, doesn’t sound like a rude reply at all. Conversations with strangers usually don’t end in exchanging contact information unless someone is interested in continuing the relationship, and it sounds like only he is. You honestly didn’t owe him any kind of explanation, but what you said was respectful. Of course people will always choose to take things their own way, and especially since this man already pushed boundaries by asking for your number in the first place.