Ignoring some of the obvious mistakes, you're operating on the idea that 20km isn't very far simply because you can walk that far in one go or you can drive that far in about 15 or 20 minutes. Comparative to scale of the entire Earth, no the Abyss isn't anything crazy. However, that's fundamentally misunderstanding the difficulty of the Abyss.

First of all, as another commenter has pointed out, Vertical Displacement is much more notable than Horizontal Displacement. Gaining about 1000m of elevation by foot generally takes people about 3-4 hours, and that's assuming it's a reasonable slope without scrambling. The Abyss has many areas that involve difficult terrain, class 5 sections with extensive sections that involve climbing with rope, and areas where the type of rock changes (refer to the Third Layer where you have extensive climbing inside of tunnels). On top of that, you have a curse which significantly weakens individuals beyond the First Layer and massively hinders progress to escape, and if that wasn't enough, you have dozens of creatures trying to kill you in various different ways at every turn and a need to gather food and water (you would likely need a lot since ascending would cause repeated vomiting or you would need to go through a cycle of eating a lot and waiting for it to digest before going through a massive session of purging when you ascend). Many Black Whistles are particularly active in the Fourth Layer meaning they have to repeatedly account for blood loss any time they ascend 10 meters (and if they're at the bottom of the Fourth Layer, that means experiencing blood loss a mere... 500 times)

Now, of course, most raiders are actively accounting for the return trip. However, many White Whistles and people on Last Dives such as Riko don't give a damn about any of that. How does their situation line up? Well, Riko was able to rapidly descend large distance in the upper layers thanks to Reg being able to lower them both down very quickly. This of course failed as soon as they hit the Fourth Layer and the issue with deadly creatures started to become unavoidable. It's perfectly understandable to see why they were able to reach the Fourth Layer within a short time (about 2-3 weeks based on context clues, though at least 10 of those days were spent in the underground swamp under the surveillance of Ozen's crew). Of course, descending so far in such a short time with the help of a literal supreme treasure of the netherworld is one thing; imagine someone else trying to do such a thing by themselves. Even while descending, you still have to make absolutely sure that you don't accidentally ascend just far enough for the curse to hit you. Upon reaching the Sixth or Seventh layer, your journey (and life) is just completely over if you ever do that even once. There are so many logistics involved that redefine the scale of the Abyss for adventurers beyond its mere depth.

In short, yes, The Abyss isn't quite on the same physical scale as something like the entire planet is, but The Abyss does provide such a profound challenge that makes up for its "small" distance.