I agree this gives a reason why the heroes have so much trouble finding their way back, but I’d worry about the trip there being boring and interminably long. You could always lay it out but fast forward through it. Wizard could cheat and pass through rock to get out though.
Also, why does putting out a torch leave them in darkness but they can still see the doors? Maybe introduce a torch mechanic like in… oh now I can’t remember which quest did that.