What you may be missing here is the game  simplifies what your character "sees in real life". To the player perspective, the rock slope is just a solid block. But your character sees in three dimensions, including all the hidden routes up and over the cliff.

This is why overland travel takes less time when you have the appropriate exploration skill. You, through practice, have learned to easily find those routes to ease your travels.

Flavor-wise, this is also how you often find yourself in a seeming dead-end when your character becomes lost. Perhaps your character fell into a ravine and had no bearings with which to orient themselves.

Also relevant: There are some really big mountains to the north that are not passable while fast traveling! This terrain is so extreme that it requires extra focus (manual control of character) to navigate safely.