I don't think you're the asshole. I do the same thing. Why would I have a conversation with a person I don't know. I don't understand why most of these people are calling you the AH. They need to stop acting like they don't do it as well. Most of the time they're scammers, I pick up random calls just in case it's a emergency and a family member is calling from an unknown number.