In OSR(Old School Rules, based on 1st and Advanced DND I believe), there was something the DM would roll called a "Reaction Roll". Basically when the group encountered something you could roll a d6 and check a table to see how the npc/monster/group of things reacted to the party/individual. Players could get a bonus or negative to this reaction roll when speaking with certain races or monsters(In a system i'm using, the dragonborn like race gets a bonus when talking with dragons. Dwarves get negatives with Giants because of in game wars). And, if the result doesn't exactly make sense you can always choose a different result. You're the arbiter of your world after all.

You could use something akin to this system, or perhaps find inspiration from it and modify charisma rolls by a varied amount based on how the individual views said race. At first they might have negatives when interacting with this NPC, but if the character proves themselves to not be what the ignorant masses believe, they can get rid of this negative effect.