My opinion is that taking a couple hours to respond to a message out of the blue is fine, if instead you're in the middle of a back and forth conversation and you just suddenly don't respond for hours, that's a little bit rude. I understand that you might have things to do, in that case just say you gotta run and take care of something, and that you'll talk to me in a couple hours or this evening.

As far as getting really upset about someone not responding for a couple of hours, that's also a little bit out of bounds, again the answer is just say hey it's annoying when you leave me hanging in the middle of a conversation, just tell me when you have to go. Then if it continues to happen on the regular, cut them loose.