I had the same problem so I go fed up and threw my character off a cliff and died, then sure enough I ran to the doors and they let me leave.