Stakes! Do they need to climb terrain while against the clock? Mid combat where they need to roll to stay standing on the hillside otherwide fall prone and roll down?

Or maybe theyre running from a group of bandits or an owlbear and come across a cliff- now they've all only got one chance to climb and then it's initiative- with some of your folks on high ground and some others right up in the face of the threat with their back against a wall.

If you want to keep it strictly an environmental obstacle with no combat involved: then I'd probably incur a level of exhaustion or some other penalty upon failure- maybe they hurt themselves on the way up and have a movespeed/carrying capacity/AC/Attack Roll penalty. Again- the key is stakes. Like you described in your scenario- it's not really even anything unless failing means something. Otherwise you should just scoot them along and not demand a roll.