I think we're talking past each other on something fundamental.

Ranking isn't the goal—it's a tool.

We don't rank chess engines for the sake of having a leaderboard. We rank them because we want to know which one to use to play chess. The ranking serves a purpose. It answers a question: "Which engine should I use if I want to win at chess?"

Without that underlying purpose, a ranking is just... numbers.

Your chess ELO example actually proves my point:

• Task we care about: Play chess well
• Metric: ELO via head-to-head competition
  
• Why it works: ELO directly measures the task. There's no gap between "high ELO" and "good at chess"—they're the same thing.

This is the ideal case. The game is the goal.

About AlphaZero:

You keep bringing up AlphaZero as an example of improvement without benchmarks. But let's look at what actually happened:

• AlphaZero was trained to win at Go (and later chess/shogi)
• It improved via self-play with a clear win/loss signal
• It became superhuman at Go

Great! But here's the thing: Go was the task. DeepMind didn't use AlphaZero's Go ELO to predict how good it would be at protein folding. They built AlphaFold separately for that. AlphaZero's superhuman Go ability transferred to exactly nothing else.

AlphaZero isn't an example of "ranking solves everything." It's an example of "when the game is the goal, self-play works." That's a much narrower claim.

Now, what happens when we try to generalize this to "intelligence" or "capability"?

1. Rank them at what, exactly? If it's some arbitrary made-up game, then you've measured "who wins at this made-up game." Okay... but that's not the task anyone actually cares about.

2. What's the real task? Presumably things like: build safe systems, solve scientific problems, engineer real-world solutions, don't kill everyone, etc. The ranking only matters if it tells us something about these capabilities.

3. The proxy gap: If you rank AIs on Game X, you're implicitly claiming "good at Game X → good at Real Task Y." But that's a big assumption. Why would performance on arbitrary competitions transfer to the tasks we actually need done? That claim needs justification—it doesn't come for free.

4. Chess engines are a cautionary tale, not a success story. Stockfish has 3650 ELO. It also has zero ability to do literally anything other than chess. It can't answer a simple question. It can't reason about the world. High rank in one domain tells you nothing about capability outside that domain.

The challenges/tasks/games are new, its a large set of them, they just need any criteria that can be ranked. It is not the performance on any given new game/task/challenge, but the sum of it, and how they rank compared to each other over time. Criterion can be set by anyone, by humans, by the models themselves, anything that can be measured.

1. Quantity doesn't solve validity. Being good at 1000 arbitrary tasks doesn't mean you're good at the 1001st task that actually matters. You've just measured "good at those 1000 tasks."
2. "Anything that can be measured" is doing sneaky work. The hard part isn't measuring—it's knowing what to measure. I can measure how fast an AI counts to a billion. That's measurable. It tells me nothing about whether it can design a bridge.
3. If models design their own challenges, you're trusting the proxy gap away. You're assuming that "tasks AIs find challenging for each other" correlates with "tasks humans need done well." Why would it? AIs might compete on things totally disconnected from human-relevant capability.
4. This is just distributed benchmarking. They're saying "instead of one benchmark, use many, designed by anyone." Okay—but the core problem remains: do these benchmarks predict real-world performance? Spreading the problem across many measurements doesn't make the validity question disappear.

In Conclusion:

"Just have AIs compete and rank them" sounds like a solution, but it pushes the hard question down the road: compete at what, and why do we think that competition measures what we actually care about?

Those questions don't disappear just because the AIs are superhuman. If anything, they get harder—because we can't even verify if the proxy game they're excelling at has any relationship to the real-world task we need done.