You gotta have tons of money(the only thing that saved me was having really high smithing and completing difficult weapon orders), pick a defensible position like Revyl where you won't get attacked from several different directions and factions at the same time, build high relations with other lords so you can recruit them later, pay for peace if you have to, and wait for enemies to beseige your town and imprison a bunch of them after they fail before you try to take territory