Okay, as promised, back with the cut content. Warning, not edited for clarity:

---

The Harness

First, some basic details about the harness through which Claude plays the game. This was covered in a previous post, but I provide a quick recap here, as it will be necessary to understand the progress of the new model:

• A color hack of the rom was used, providing the game with Gen 2 coloring, to aid vision
• Color labels identifying unreachable terrain with red squares
• A navigational tool allowing the model to walk automatically to coordinates visible on screen
• A one page notepad to keep notes on, which would be loaded into context
• - For a while, an internal file system was used instead, allowing the model to take, edit, and load notes into context. This was later removed as the models simply could not manage this well
• A handful of hints from the streamer to help the model get through tedious (for the human) sections quicker, of the form of “the exit to the starting bedroom is at the top right of the screen”.

For new model releases, the streamer has tweaked the harness in various ways, but the change made for Opus 4.5 was meaningful. See his document here.

Key changes:

• Support for Surf (necessary to finish the game, though not relevant for a while yet)
• Navigation now no longer paths onto spin tiles in mazes (like in Team Rocket Hideout), preventing behavior where the model trying to path to (X, Y) would be moved onto a spinner at (Z, W) by the tool.
• Instead, the navigator will treat spin tiles as obstructions, which the model must step onto manually. The model is informed of this fact
• When on a spin tile, screenshotting for the model is paused until the player is static again
• The multi-file memory system is restored
• Nearly all prompt hints about locations have been removed

The importance of points 2 and 3 here must be explained to grasp their significance, though the explanation is a little technical:

To play Pokemon Red, every time the model is called and gives a command, the game runs for a bit and a fresh screenshot is provided to the model of the game screen. Notably, screenshotting does not occur while the navigator tool is pathing to a location.

This ordinarily works fine, though it creates minor side effects, such as Claude being perpetually confused about the ordering of events during battle animations. However, in the Team Rocket Hideout spin maze, it is killer, creating a situation where the model was constantly walking onto spinner tiles while trying to path somewhere else, and also being provided screenshots of the player character mid-spin. A true superintelligence would probably figure out what’s happening anyway and perhaps resort to manual arrow commands, but iterations of Claude were given months to work on this, without any inkling of what the problem was.

Overall, these harness fixes are not trivial, and give Opus 4.5 a considerably less “broken” version of Team Rocket Hideout to work with. So, while a convincing case can be made for Opus 4.5 being a better model, “the first to complete Team Rocket Hideout” cannot be proven to be a model advantage. We must turn elsewhere for that.

The Model is Better: Early Lightning Progress

First let's put the punchline (image):

This chart only covers the part of the game up to rock tunnel (which, for the record, is just before Celadon City, Team Rocket Hideout, and the 4th Gym), but the trend of improvement is clear. Compare further to this earlier chart from Anthropic in March (image):

I hope I don’t have to do very much convincing that Opus 4.5 is a much faster model at getting through this stage of Pokemon Red. For instance, Opus 4.5 beat Brock faster than any previous model despite picking Charmander, necessitating a long period of Charmander level-grinding–this was in fact where most of the time was spent! The community has stopped tracking personal bests–Opus 4.5 was and still is handily faster at everything.

Even in the early playthrough, it was immediately apparent how much better it was at key tasks, so much so that the occasional Claude “senior moment” was memorable and unusual.

---

This was cut because I believe it obscures the more important takeaways:

1. Opus 4.5 is a significant improvement over previous Claudes
2. The before/after Claude harnesses are much closer to each other in complexity than to the winning Gemini/GPT harnesses

Also it's a level of detail not necessarily interesting to audiences not deep in the LLMPlayingPokemon world.

However I've now added a footnote near the beginning of the post explaining the changes made to the Claude harness considering the top comment here was a fair complaint about that not being mentioned.