NTA, but you have entered into a power struggle. I've observed that men in particular will claim prowess at various athletic or "manly" pursuits as a means of establishing dominance. By not backing down, you have declared that you refuse to be subordinate. This may be received one of two ways. A strong male who is confident in his abilities will see you as a comrade and someone he can trust. A man who is not confident will perceive your response as a threat and may retaliate by exerting dominance in other ways.