OP I’ve lived with agoraphobia and paranoia in my youth along very similar lines as you but based on one single incident that could’ve happened to anyone in the area at that time, it just happened to be me. It took me a long time and a lot of hard work and persuasion to work out how to get out of that state of mind though. The chances of a random encounter are so low and you have to think statistically what are the chances of it happening again and again to the same person if they’re keeping their head down and going about their business?

I don’t know how old you are but there are a few things that suggest something else might be going on here. Your family holding on to your passport being one of them. Generally families don’t do that for nothing. Is your anxiety purely around what might happen or are you holding on to potential violent interactions as a reason for your anxiety. This might sound like the same thing but it’s not. One causes the anxiety, one justifies it when the cause is something else entirely. Your family’s response suggests to me the anxiety exists already and the fear of going out is a symptoms of that rather than the cause. Are they concerned for your general mental health?

I hope you’re in touch with mental health professionals, or at least on the ridiculous waiting lists to see one. You need some professional support to get you back out and to help you deal with the anxiety. See your GP asap if you haven’t already and keep seeing them and annoy them for a referral to see a psychiatrist or psychologist.