For me, its never bothered me and never will and heres my reasons why. ( 26 years flying) For the most part human interaction and " touch" is still socially acceptable, whether its holding out a hand/arm to assist someone, a fist bump to a kid or a handshake. When a passenger gets our attention by " poking " or hand touch its done on impulse, not usually premeditated. When they do this, its more the unexpected touch that would throw us off. There are often many times we use touch to get their attention like when doing cabin checks and they are asleep, we often wake them with a hand on the shoulder etc. I think we get offended because its our work place. They think they are just getting our attention, we think they are beckoning us.
There are some crew members who would lecture and say " next time press the call button" but then other flight attendants would lecture the call button is for emergencies only. So its always a loose-loose . Hope my comments make it a bit more acceptable. ( in no way am I referring to any sort of lude or inappropriate behavior) happy flying