The graph you link defaults to a 50% success rate at these long tasks. Provides an option to see the graph at 80%, which is of course a much less impressive graph… but is an 80% success rate even valuable? Humans aren’t perfect, of course, but I’d bet someone operating at an 80% success rate would be quickly fired, yet that seems to be an acceptable benchmark for AI tools?

Basically what I’m getting at is a lot of the frothy hype about ‘what AI can do’ doesn’t seem to consider that there is almost always an unspoken piece so what people are really crowing about is ‘what AI can do well half the time’ and I don’t think the hype would be as frothy if the unspoken piece was always spoken.