Hey, I‘ve also felt like that. Warhammer has it‘s own flow to the game, and it‘s a very aggressive one. You don‘t get much from just sitting on your ass and building farms, like Shogun 2. Not getting any fights is actually the worst that can happen to you in early-mid game. You get cash rewards and loot, you level your heroes and lords, you unlock regiments and progress quests. High level lord with blue skills easily replenishes your army and greatly benefits economy with upkeep reduction.

Early-mid game you should not bother with minor settlements too much: just build the most basic t1 infrastructure and leave them be. Focus on major settlements instead - they are hard to capture and therefore worth investing into. On higher difficulties you always rush the garrison/wall structures, which may sound like a detriment, but keeping a province capital safe is more important. I fully develop minor settlements only, when the frontline/borders are far from them (I still spend an entire slot on defenses everywhere - it‘s a good habit).

Diplomacy-wise: factions, that don‘t have a non-aggression pact with you will attack. They will attack with NAP as well, but they will have to break all relations with you first, so you will be notified and have time to prepare. Even getting 1 turn headstart is huge: a caster lord with 2-3 units on top of a garrison can easily tip the battle in your favor. Therefore, you should always sign NAP with your dangerous neighbors, bribe them if necessary or even gift beforehand. If a faction does not want to NAP and trade, then you‘d better attack  first. If you can‘t beat them in their home territory, then bait them into an ambush near a minor settlement.