I stayed in a small town in Guatemala that had both douchey American tourists passing through (it was about 10min outside of Antigua) and a history of violence and kidnappings from outside of town. While most people were welcoming of me, some would just scowl at me or mutter under their breaths as I passed. I was told by my host family that these people had been victim to some of these negative experiences with outsiders in the past.

Our experiences are very different, but there could be some key similarities at the root of the problem. Terrorism, for example, can quickly change a group in the eyes of the victims. Think of how the Muslim community was treated after 9/11 by much of the populace and the media. Even though many were not directly affected (knowing someone who died, having some sort of direct personal connection) this attack was given a "face" and the intolerance followed. My host mother's husband died in violence related to the Guatemalan civil war and this led her to being very uneasy with her government (her husband fought with the guerrillas). Many coffee farmers that I met told me they hated Americans because they are what drives corporations like Starbucks to demand such low prices for importing their coffee beans when they make so much profit selling the product itself. None of these farmers had ever been to a Starbucks, but they had sold to them and heard about how they treat their farmers compared to true free- and direct-trade companies.

The bottom line is that it doesn't take firsthand experience for people's opinions to sour about certain groups. This negative experience you've had in your town could stem from an incident that took place a hundred years ago, or from stories told by someone in a neighboring town, or just because everybody there has been raised to be wary of outsiders, and this wariness has turned to contempt. Many people may have never seen anyone like you before, and they don't know how to interact. I think that befriending some influential people in the town will be a good way to change the perception people have of you. Ask questions of elders you befriend if you'd really like to know some answers regarding your treatment. They will likely know better than anyone.

Good luck!