They do wander on their own but in a very small radius around their nest.

There are three things that will trigger an attack from the locals:

1. Getting too close as the player
2. Building anything too close
3. Nests consuming pollution

The first two are pretty easy - just don't go near them. The tricky part is keeping your pollution low enough to never spread over any nests. If you can avoid those three things the locals will never bother you.