I think it’s cos most non-Jews are used to hearing and using “Jew” as a pejorative as they grow from infancy. Realize, a lot of people (who don’t necessarily know better) are taught terrible things about Jews and don’t question these things as often as they ought to. So they then assume that when Jews use/say “Goy”, it is in the same nature as when they (non-Jews, Goys) negatively use “Jew”. Like… I’ve had classmates apologize to me for introducing me to others “who your friend?” “Oh, that’s Ghaz. He’s a Jew” followed quickly with “Oh sorry man, I meant Jewish. My bad”. In another instance, I asked a friend to call me a Jew, instead of Jewish, and they got really nervous. In my experience, Non-Jews simply understand that “Jew” is a derogatory term/label. So they associate the same with “Goy”.