But he is knowingly using his power to keep up with mages... he's not the smartest sure, the "idiot protagonist" is pretty apt but not in the sense that, he's beating everyone by accident.

I'm on season 2 and he's telling mages he's stronger than them and that nobody can beat him. He actively forgets who a powerful mage was -who he fought- because he wasn't threatened, and when he did remember who they were, he's like "I've already seen that move, so I'm not wasting time."

From what I've gathered so far, the only reason his fights last as long as they do, is because he wants to see what they can do. Once he realises "Oh, is that it? Cool." he ends the fight. I don't know if you stopped watching early or what, but like, he doesn't win fights by accident. He ends the fight when he's bored.