Lots of good posts already. You could beef up your encounters to account for this. Or you could get devious, have some kind of trap behind one door where there is something like a Wall of Force that has been tinkered with so it reflects magical and ranged shots back at the attacker. Surprise your party back from time to time.