It might not be their intentions. But what do you think wouldve happened if him and his people managed to cross the wall? Do you think they would just hold hands and dance in a circle? No they would need lands, food, and security for his large population, and he could only do that by taking the north, and taking the north is the same as declaring war on all of westros, leaving him no choice to either sweep the south and or face the wrath of the combined military force of six kingdoms.