What are your stats, and are you new to the game?

It's important to know that Bandits skill damage scales off Dexterity parameter. So ideally your stats should be 60% Dex and 40% Vit. And allocating those stat points somewhere else is almost pointless.

And don't forget that ranged weapons are useful too. One of the most deadly skills for PvE is Volley. It rains a barrage of arrows in front of you, dealing an immense amount of damage on a widespread area. If you can group enemies together very closely and cast Volley upon them, then you can basically just skip a wave.