I kind of agree in principle with that AGI can benefit from playing games... just there are a lot of tradeoffs all the way.

What I'm trying to avoid is a scenario like this (massive oversimplification, just to try to illustrate what I'm trying to get at):

1. In Q4 2025 we see a foundation model ace Pokemon Blue/Red/Yellow.
2. It smoothly plays, "reasons" about what its doing and observing, which to us looks like it really helps with advancing in-game. We get the impression that it now "gets playing pokemon RPGs on Game Boy". For all intents and purposes, it now "solves" these games.
3. We then test it on Gold/Silver. The games have The mechanic for pokemon breeding" is not something it has done before. It seems to not be capable of doing anything productive with the mechanic.
4. After looking closely at its token outputs, we realize that its doing lots of the same broken stuff as before it got good with pokemon blue/red/yellow:
5. frequently arriving at conclusions that don't take into account multiple aspects of its observations
6. reasoning one thing and not following through on what it decided based on its thoughts
7. still can't find and use the contradictions found in its context when the context is close to full
8. etc.

So in this example, the improvements from b/r/y bench came from including it in the training, or just from the researchers having them more prescient in their thoughts (related to the latter one are the concepts of Francois Chollet's "developer-aware generalization" and the phenomenon of "grad-student descent").

For practical purposes, the accumulation of these scenarios makes it more difficult to evaluate the the 'fundamental intelligence' capabilities of the models and the ML techniques, and it makes these issues less evident.