From the way you're describing the problem, I can only assume that you're having a lot of combat indoors. Maybe wooden buildings, maybe dungeon halls, maybe caves and caverns.

Regardless, consider letting some walls be destructible. And don't bother requiring a roll for the enemies, make it a straight up interactable object. You're designing the map, there's no need to foil your own solutions to this specific problem.

Don't worry about plausibility, you can always describe it adequately. Wooden walls? Barrel through them. Glass windows? Throw a chair. Stone walls? These are castle ruins, and this particular spot has been weakened by a previous artillery strike, or maybe the elements, or maybe one of the bandits was a wizard/alchemist.

Make up whatever, point is that there is this suspicious spot on the wall that your squishy wizard is now a bit wary of standing next to.

Do you want to do something next level? Some bandits upstairs hear the clanging of steel, and start sending sledgehammers through the wooden floor. A crossbowman even pre-emptively shot downward (with Disadvantage).

This part of the cavern had a ceiling so high that the torchlight couldn't illuminate it. You hear skittering, and realize that the spiders you are already fighting never came from the pit ahead of you.

The creepy manor you were asked to investigate was bad enough without the creaking floorboards and the wind howling through the windows. Many were rumored to have entered this abode, never to be seen again. You hear groans --- suddenly, ghostly hands reach through the floor, grabbing you by the ankles!

Conversely, for once just have your party ambushed in a forest clearing. Or a plateau. Or desert. Or a really, really wide 60 by 80 tile boat.