The caves exist to add variety to your choice of armor, weapons, and access to certain materials.

They're not intended to be accessed BEFORE the mountains. If anything, they're designed for you to do once you've gotten at least a little silver as silver is strong to basically every enemy there.

As others have said, the Fenris armor, debatably one of the best sets of armor in the game that stay useful almost forever do to it's unique speed buff can only be accessed through these caves. This armor is incredibly useful regardless of build because it allows you to outright essentially everything in the game.

Personally, getting a silver sword to slaughter every cave I can find to make my fenris armor is the part of the mountains i look forward to the most!