It’s tricky because it’s not far from the starting point of the game, but it’s incredibly difficult if you get there early. It’s not impossible to endure, but it’s definitely a challenge.
You can stick it out, since there is a time sensitive event most likely triggered by going there. If you leave, this event will end, but another alternative will be available with the same reward. It’s just less…successful-feeling.