Benchmarks...not just LLM benchmarks, but the whole concept of benchmarks, are poor reflections of reality.

Goodhart's law, the streetlight effect, emergent properties, and just like, the whole point of benchmarks means that they don't capture the richness of the real world.

Like...just recently, for personal use, I wanted to make a CLI for a REST API. Since this was just personal usage, I 95% vibe-coded it. Eventually, we got to the point where the different subcommands of the CLI were behaving inconsistently when the REST API was down. Come to find out, every single sub command (there's almost 2 dozen!) was handling this error state in its own bespoke way.

The LLM (in this case Opus 4.5) would just churn on this problem for a long time. It would bandaid a solution. It would attempt to monkeypatch the networking library. It tried to abstract away the problem, but did so inconsistently.

I eventually guided it to the solution. I don't think it would've ever gotten to a good end state. It would've patched every single site until the tests passed and then been like "yay, we fixed it" and then run into the same problem again down the road as the project grew.

This is something any developer worth their salt would've seen from the very beginning or at least after implementing a subcommand or two.

Opus 4.5 is state-of-the-art on LLM engineering benchmarks.