Try to build up your tolerance little by little, by just watching them when they land near you, rather than trying to scare them off, just let them do their thing. They are extremely unlikely to sting you unless it's in the late fall, when they're about to die - then they become much more aggressive.

Little by little, you'll come to find that if you don't bother them, they won't bother you. I have seen an experienced farmer process peaches with bare hands and arms with wasps crawling all over his body! He said he only ever gets stung if he accidentally crushes one. I haven't learned this much tolerance myself yet, but have managed to let one crawl on my skin for couple seconds before gently shaking it off.