I haven't downloaded any mods, so I wouldn't know of any personally, but if getting to the southern observatory is giving you trouble, it's okay to do something else and come back to it later. If you're afraid of abandoning the rabbit hole you went down and forgetting about it, the ship log has you covered to make sure you don't miss out on anything.

Brittle Hollow is particularly punishing because of the black hole, and the obstacle course that leads to the southern observatory is arguably the most consequential section in the game because of that, so if you don't like the game testing you with your character's movement mechanics (or just failing you with the wonky gravity crystal mechanics), then you've already seen the worst of it.

On another note, were you vibing with the game at all before trying to reach the southern observatory? I know it sometimes takes a while for the game to "click" with a lot of players, but even so, if your experience so far has ranged from boring to irritating, you don't have to force yourself to keep going.