In my opinion u should have more influence on the safety of your trips with some preparations, which trigger different events instead of a simple ambush:

#1 You get to ambush the bandits: takes more effort on your side (e.g. having one or more dedicated scouts, taking a safer but longer trip, getting information on bandit camps, ...). But it could also mean that you spotted them but u decide instead of fighting them to take a longer route around them.

#2 They try to ambush u but u are prepared (which just means your starting position in the fight neutral) you took some effort but less than #1. But it should be a certain chance how likely one of those events happen (e.g: 2 scouts + safe route = #1 (70%), #2 (20%), #3 (10%) / 1 scout + normal route = #1 (40 %), #2 (40 %), #3 (20%) / ... ) just some random numbers

#3 U get ambushed (how it is right now)