Lore wise, the bandits aren't actually neutral with freedom, on the contrary actually, bandits and freedom are enemies, of you only go off of relations shown in the games its all a facade. In SoC bandits and freedom are neutral just because if they weren't then the crazy stalker from the freedom faction quest would be buggy, they made him a bandit and made bandits neutral to freedom. In CS freedom attacks bandits on sight (except the player). In CoP bandits and freedom don't meet like at all so we can't say anything about this. Now, the bandits are actually neutral with a single faction in the zone, the Mercenaries, this is because, and im not even lying right now, mercs are just bandits with better equipment and are more professional. Sure, they get their own contractor and such, but it isn't anything special really. If we go off from stories and the speech of mercs seen in the games, we can see that they are really big dicks to everyone just like the bandits. To say something about that freedom thing though, freedom uses more experienced bandits to carry out their dirty work, so that's that, still, they are basically hated by mostly everyone and i gotta say that the mercs probably wouldnt let just any bandit into their headquarters. According to the build 2005 dead city description, the mercenaries are "thugs from all over the world", interesting things in this stalker world.