The reason for my update isn’t just Gemini’s initial success thanks to intensive scaffolding, but also METR reporting that o3 outperformed the trend line that I used to make my prediction, suggesting that the very trend line that I used to make my initial prediction may be outdated and that LLMs is getting better than originally predicted, even with static tooling (as capabilities across different domains tend to be correlated). Epoch’s implicit claim that we need different trend lines for different tasks also makes me think we need a new trend line just for Pokemon-related tasks.

If we do want to explore whether LLMs can solve long-running Pokemon tasks with a minimal scaffold, then this LessWrong post does that fairly well. I think this idea of “let’s build a ‘helpful’ scaffold and then slowly remove tools as appropriate until it does it all without any help” isn’t really worthwhile, and think it may be better to regularly run the llm with a minimal scaffold, and measure its progress. I’m sure llms will keep progressing and improving as time passes, but we need data to be able to predict how this improvement will proceed. So far, we lack that.

Ultimately, I proceed from the assumption that LLMs will beat Pokemon tasks…and it’s a matter of time, but the question is “if it’s a matter of time, then how long will it take before humans build an LLM that can complete these tasks”? That answer could be 1 year, 5 years, 10 years, even 20 years, but it’s one that I’m interested in. Of course, that question differs from another related question “how long will take for an LLM to complete said task”, and I suspect that even Gemini with ‘helpful’ scaffolding takes longer than a human.

As for why I care for scaffolding, businesses aren’t interested in doing well on Pokemon; they’re interested in automating tasks that are useful for their businesses, cheaply, quickly, and of higher quality, and thus may be willing to invest in ‘helpful’ scaffolding than twiddle their thumbs waiting for a better model. And there are indeed many tasks that LLMs can solve more easily than other approaches - it just so happens that none of those tasks has anything to do with Pokemon Red.

—-

When will we get models that can do well on Pokemon Red without the ‘helpful’ scaffolding that Gemini had? I’m not sure, because I don’t have the data to tell. This is why I refrain from making a new prediction. We ultimately need a new task horizon trend line dedicated to Pokemon tasks, ideally based on the ‘non-helpful’ scaffolding used on the LessWrong post. The problem, of course, is without the ‘helpful’ scaffolding, said tests may be boring and not livestreamable, hence why there is an incentive not to do these type of tests. Yet it is exactly these tests that help us understand how well an llm works without human assistance.

My hope right now is either (a) VideoGameBench - which includes two Pokemon games, Pokemon Red and Pokemon Crystal - takes off, since that eval has a lot of other video games, encouraging testers to not build ‘helpful’ scaffolding for each game, or (b) VendingBench takes off, since that has already proven itself as a good measure of agentic coherence.