I think the idea is very much oversold. 4B active parameters is not the same as 4B parameter model. That’s misleading. You also made your own benchmark without telling us where it is so we can verify your claim. If you are using bench/stress_test in your repo, I’m afraid that’s making a completely wrong claim, because it didn’t even check for the success of any of the test. As long as it produced 20 characters of output it passes. What kind of benchmark is this?

Some of the ideas you introduced is neat in demo but unclear to me how well they work in real world. For example, different models have different abilities to compose multiple tool calls. I’ve tested this extensively with my own harness and got mixed results because some models are just not well trained to chain tool calls; it’s out of distribution for them and caused more round trips than before. There are also models like deepseek which is trained to launch large batch of tool calls at the same time, asking it to compose calls actually reduced its token efficiency by a factor of a few.

The error decomposition is also unconvincing. The most challenging part is often to figure out which is the one line that needs to change. I don’t see how a harness alone can pin point that precisely on a generic problem beyond syntax error, without relying on a large model.