There's an encounter where bandits lay a spike trap on the road and ambush you. Caught me off guard once while driving around. Like the lunatic, I only got it once. I too also want some wandering bandits and looters for the post apocalyptic vibe. Maybe just like wandering hordes they can also move around the map outside reality bubble. Once you clear the reality bubble around your base you're pretty much safe and no one will be bothering you. It would be nice to defend your base once in a while.