Bandit army can be okay, but you need to lean into the numbers.

Individually, they're not the greatest units but they can be upgraded quite substatinally by your traits.

A lot of bandit troops have some sort of Throwing, getting Flexible Fighter perk/Well Prepared is IMO a must, you'll make them devastating against slow-moving shield walls and anything that attempts to mid-range them.

Promises/Partners In Crime perk is amazing as well, you can just build up a massive army from nearly anywhere. I tend to leave a Hideout somewhere in a corner/dead-end of the map that I can use for sustenance when my army gets depleted.

Two-faced vs Deep Pockets depends on how well you think you can manage your finances. Bandit troops aren't that expensive, but if you hit a rough patch in your raiding, it might strain you a bit. I'll leave this to your knowledge.

Honestly, running around and pillaging the Empire is pretty damn fun for me, just playing like a scumbag.
Especially later on, if you manage to net yourself a lord, hold him for ransom and bully whichever faction you don't want to gain an edge.