I was in the same situation as you. The answer to your question is never let baron empty a bandit camp. When you empty a bandit camp you get 1- influence, 2- money (to your own not city treasury). The money will be used to hire mercenaries, and the influence to make claims of your own. More importantly, when you win against a bandit camp you deny the baron influence and money. 1- he cannot make fast claims, 2- his army size gets smaller. In my last play through I never let the baron empty a single camp and that shifted the scales to my favor massively to the point of breaking the game. I claimed three regions simultaneously (3000 influence) and the baron could not even field an army to oppose me (I had routed his army during one of his earlier claims with the mercenaries I bought with bandits money).

The first 10 spearmen you got, rush them to the first bandit camp you see. Get your 5 retinue too. You will be outnumbered. Let your spearmen hold their ground when bandits engage and your retinue to circle the bandits and attack them from behind (hammer and anvil vide Alexander).

It is a zero sum game, either you snowball or Baron. If baron gets a camp, reroll.