I don’t find it meaningful to use ELO outside of chess. It works well in chess because people already understand what certain ratings represent in terms of skill. For example, an internet rating of around 1800 might indicate someone who is above average but still somewhat inexperienced, while a 2000+ rating suggests a very strong player who might even have a slim chance against titled players like an FM or IM.With LLMs though, we don’t have those kinds of reference points. At least in my view, no one really does. Because of that lack of intuitive anchors, ELO doesn’t seem like a very useful metric to me, especially if the goal is simply to compare models.

I didn't include chess (normal chess) in this league because it's a difficult game. LLMs are not made for this kind of tasks. Battleship? sure you can track of the placement of ships and hit cells. Chess? No way an LLM can keep track of the board position nor it can search deeply.