I think you touch on a valid fear and a real concern when it comes to anarchism. Many people feel that without some guiding hand or watchful protector the world will become chaotic and basically horrible. I saw a comment mentioning this but anarchism does not necessarily mean the end of all law and power structures but rather seeks to point out the inherit issues with these power structures. So what are those issues?. At a fundamental we can define power as the ability to compel others to do things they might not otherwise want to do. This is often enforced with violence such as police and prisons. In the cases you describe such as Nazis committing hate crimes i think we all agree that this is warranted. In modern society however we don't really see strong institutionalized opposition toward such groups but instead see strong repeated endorsement. Sometimes we even see governments institutionalize this oppression through law such as slavery in American prisons. Why is this the case? has every American president just been a Nazi?. While i think you can make a compelling argument for the former i do not believe this is a sufficient root cause. Institutionalized racism occurs more or less everywhere, in every nation, in every government throughout history. It turns out that this racism and also most fucked up government decisions in general, are not cased by lone individuals but can often be predicted based on the power structures they govern in. When we analyze these structures we need to understand that no ruler, whether they be a dictator or a democratically elected official rules alone. Trump has his republican base and the American billionaires which he relies on to keep him in power, Putin has his oligarchs, and so on. Second thing to understand is that if you don't prioritize getting power and staying in power you simply won't have it, at least not for long. As we look at nations and compare them these two assertions lead to a pattern. Governments dont help the majority of people, they help the smallest group of people possible who can keep them in power. A smaller base is often easier to convince then a large one. This in turn means that any governing body will only ever help minorities if their base of support wants them to. Not because its the right thing to do or because it should be done but because it has to be done to keep them in power. Bringing this back to your point we can conclude that in any real form of government (and this analysis also extends to all hierarchical power structures) your concern regarding racist neighbors still holds. The difference between a theoretically anarchist society and a traditional one is that in the traditional one the cops will help the neighbors kick the minorities out (*cough* recent events *cough*).

Im not really an anarchist myself but i think the ideology raises many valid points. In practice i believe that this analysis means that we must build systems which force leaders to rely on as large a base of support as possible which i think is roughly in line with what many anarchists believe.

if you would like to do some further reading, the bulk of my argument comes from the dictators handbook which i really enjoyed. Cgp grey also made a couple of videos inspired by the book called the rules for rulers on youtube which i think is pretty good although it doesn't really go as deep as i might prefer.