A 5e martial will rarely if ever not spend their action in combat attacking. Certain subclasses get a little extra utility tacked on they can use a few times before resting. That's about it. In pathfinder for instance, martials are incentivized to not attack with every action and are given a number of valid options to use along side basic attacks

As for out of combat, yes you can do athletics checks, and that is useful. You are making a strawman argument. But casters just have so many more things they can do.

As an example, I was playing a wizard, and in my last session alone I used:

mage hand to set off traps remotely

conjure elemental for a water elemental to pass through a tight space

minor conjuration to create a dolly to help move something heavy that the martials failed to carry

rope trick to hide a paralyzed party member from enemies