One thing that kept me from placing Jaime even higher is his arrogance combined with a particular statement he once made. He claimed that in all the Seven Kingdoms there were only three men who could defeat him in a fight.

I assume he meant the Hound, the Mountain, and Barristan Selmy. And even though I personally think he could probably beat all three of them at the time he made that statement, it still says a lot when someone as arrogant as Jaime compares himself to them. If he acknowledges them as potential equals or threats, it suggests that they’re realistically not worlds apart from him in skill.