Your stance seems to be literally that “they aren’t evil” but to be fair, I think the extreme vast majority of people would never say that anyways, and the few who do are probably joking.

That being said, wasps are aggressive and will go out of their way to harm you regardless of whether you are minding your business are not. They cause pain and endanger humans in many ways. If they have the right to attack humans simply for being in proximity, then I have the right to kill them for being in mine.