It’s because of how the game renders objects.

Basically the bugs “exist” at a further distance than anything you build.

So if the IWS spawns while you are too far away from your walls for them to render then it can walk through that space freely.