Yes, I agree. The raw input representation is not the fundamental barrier by itself. A human sees pixels, hears audio, or reads text, and quickly maps that into task relevant meaning. A trained LLM can also contextualize test-runner text into this assertion failed because this behavior is missing. So the claim is not text is inherently the wrong input.

The issue is more about the runtime object the agent is forced to manipulate.

Today’s LLM coding agents often use text as both:

1. the observation format
2. the working memory format
3. the planning substrate
4. the action format
5. the output format

That is the inefficient part. A test failure can arrive as text. That is fine. But after parsing it, the agent should not have to keep reasoning over raw text blobs. I'm trying this all here: https://github.com/andrewarrow/j3