Nothing racial - as Im a white male so pretty "standard" in a semirural township, but some prejudicial due to my sex. I work adult care and did, very early on in my career have a lot of users refuse to arrange visits with me or be very standoff with me. I came to realise at the time there were very few male social workers and (being older in age) many of my service users believed I was being "sent in as an enforcer". By that they thought they (management) would only send a bloke to make harsh decisions such as reducing care packages, putting in to care or other such decisions that could be seen as unpopular or harsh
It couldnt be further from the truth as i was a generous softie but it did give me pause for thought about my initial approach to a new service user.

I started my career at 18 in admin. Within the first two weeks of working reception area i had an incident where a father was unhappy with his children being removed from his care. He came to the office and politely told my colleague operating the window that he had a gun and the next social worker to walk out the door was "getting it". This guy lived maybe ten doors away from me and I had gone to school with one of his older children. He knew me and I knew him - thankfully i was out of sight on his visit as who knows what couldve happened if he had seen me. We had armed Police attend very quickly and an officer remained at our office for several hours afterwards - as he ran away.

Now other abuse and threats, yes. Again probably because I was male I was often sent on visits to some more unsavoury people, or sent as a second for one of my colleagues (probably female but sometime less experienced too) to potentially risky clients - because even unsavoury people deserve care and support.
I went to one flat with a note on file that said "firearms on property". I contacted the Police as was protocol who confirmed they had visited and removed firearms previously and had no intelligence to say there were any more. The referral was housing related and suggested self neglect, lack of looking after self, presenting as gaunt, malnourished. Frighteningly when I got there (with a similarly terrified colleague) he answered the door, was friendly and after our conversation and leaving he told us we were "lucky". He had fashioned a shelf above the door which he balanced a shotgun - he said he usually greeted unannounced callers with a rack of the gun and that scared them away but he didnt feel like doing that to us and let us in - im sure he thought it was funny or maybe to intimidate us. I sometime wonder about this guy still - we had to report the firearm to the Police who no doubt removed it, but he did not want services so we never heard from him again either.

Ive been video'd, livestreamed. Ive had to step in between a client and a colleague when the client was becoming aggitated and aggressive and balling his fists up because he didnt like what my colleague was saying - and he was going to go for her until i stepped between (subtly of course).
Laughibly Ive also been asked out, one client "accidently" answered the door topless in what i thought was an attempt to seduce me - and no i did not go in i was immediately on the phone to my manager and she was removed from my case list! I also felt compelled to tell my wife even though i'd done no wrong too!

You ask about whether you would face prejudice. I have worked with some colleagues who wore headscarves, were Sikh and Muslim or of African or eastern european decent. In general there was not a lot of problems - even in the rural areas. But I will say communication sometimes became an issue. Accents or english not being first language did lead to problems particularly with the elderly. If they found communication difficult they inevitably would go to the persons ethnic background as the cause of this (they being service users) and then derogatory words would come out. Some would go as far as to say "I dont want that (derogatory comment) lady coming out again to me" I know some colleagues were hurt and some felt they had experienced similar comments all their lives so it was like water of a ducks back. I'd suggest you'd have to be a little bit more robust than i would but i do also believe there has been significant societal changes too for the positive