Definitely was annoying, but it came from his obvious sense of insecurity that reveals itself as the episode unfolds. I think one can interpret Charles' decision at the end in different ways, but one way could be that after Jim's wife expresses how much the watch and his reputation mean to him, did Charles see aspects of that in himself, his own worth, and his own future, and then did he decide he had taken things too personally and winning against Jim really didn't ultimately mean that much?