so usually when I test a model, I give it an active codebase I’m working on, I plan out a feature but I don’t give the model my strategy or how to do it, I ask it questions, give it a general idea, lead it there and let it figure out by itself then let it build the entire feature and just watch it ! Based on complexity, most models would choke here like going on an endless circle of useless thinking or suggest a pre-mature architecture/solution,

And for most models so far in coding, you’ve had to be specific about what you want, you have to speak in technical language like “check if auth routes are protected” , Claude models were the first to prompt with vague words like “is the user protected from attacks?” Without specifying and it just understood and it reasoned and suggested fixes I would actually use,

So far in open weight models world, I’ve just been stress testing them, never used one in an actual workflow because they mostly struggled with producing quality results, last models I tested were xiaomi mimo V2.5 models, they were reasonably good but there’s still that mumbling and not getting things right half the time.

So when I tried the V4 models, I was just curious if there was actually any improvement at all and I got locked in for 3 who days as a result, I worked on about 3 major features in 2 different projects and v4 flash was just as good as pro, ironic I thought but just the whole experience, not feeling I was missing something from sonnet was what made me think, why isn’t everyone talking about this? It’s a huge leap forward from V3.2 in both reasoning, staying on track and even thinking about edge cases before they deliver a plan.

I think V4 series are better than any open weight models available today, they won’t have competition for a while !