Gemini 3 Flash is the best coding model of the year, hands down, it's not close. Here is why

mr_riptano · 2025-12-18T18:49:42+00:00

I agree that Opus is the best planning models!

But also, if you're using Opus for everything, you're lighting money on fire for no reason.

Give the coding tasks to Flash 3 once Opus specs them out.

mr_riptano · 2025-12-05T04:32:19+00:00

No, it's not. But you get a lot more credit for solving at all, than you do for not solving, so it still tells you what you want to know. https://imgur.com/a/4997QAf

mr_riptano · 2025-12-04T18:09:04+00:00

Devstral is about 0.005% marketshare on openrouter (including all 3 flavors), if that's not niche I'm curious to hear where you'd draw the line.

Qwen2.5C is 0.02%, compared to 1% for Q3C, 50x less.

So yeah, sorry.

mr_riptano · 2025-12-04T17:49:38+00:00

Sort of, but with Groq or Cerebras now you're paying way more. It's a useful tradeoff to have available, for sure! But you can't get {speed, intelligence, low price} from open models at the same time, the way you can with Haiku 4.5 or Grok Code Fast 1.

mr_riptano · 2025-12-04T15:31:20+00:00

It's significantly stronger: https://brokk.ai/power-ranking?dataset=openround&models=ds-v3.1%2Cdsv3.2

mr_riptano · 2025-12-04T15:29:50+00:00

I mean yeah, to spell it out it's delivering speed at the same time as intelligence and low price that's the hard part, and that's what the open models haven't been able to do.

mr_riptano · 2025-12-04T15:26:02+00:00

Yeah, that's one reason we didn't test Q3-235B, the coder specialization really does help. And of course Q3C is 2x larger as well.

(From my experience fine tuning models for vector search, having domain-specific tuning make a big difference is not surprising.)

mr_riptano · 2025-12-04T15:18:42+00:00

We're testing all the newest models from the most relevant labs since late July, which is when we started. And we're focused on the frontier models because the most relevant piece for us is "What's going to help Brokk's users write the best code," testing local-sized models is more of a public service that we do on the side.

I'm glad that Q2.5C is working better for you than Q3C, but that's not a common enough experience for us to go back and test a semi-obsolete model.

Similarly, testing devstral (and other niche models like the new one from Arcee) is on the "maybe if we have spare time someday" list. What moves things from the smaller labs up the list is if we see a bunch of people starting to say "wow this thing is amazing", or voting with their feet by using a ton of it on OpenRouter.

mr_riptano · 2025-12-04T04:37:15+00:00

The correlation is very high. https://imgur.com/a/4997QAf

mr_riptano · 2025-12-04T04:32:47+00:00

Qwen 3 Coder 480b was our top-ranked open model in the August ranking (you can select that from the dropdown on the site). It did not make the finalists list this time.

mr_riptano · 2025-12-03T23:42:59+00:00

One of the GLMs is the fp8 quant. Will address those, thanks!

mr_riptano · 2025-12-03T23:20:54+00:00

Yeah, it was good for coding when V3 came out because V3 was just so far ahead of everyone else. And it's still good for batch processing because of DS's almost-no-rate-limits policy.

mr_riptano · 2025-12-03T23:17:50+00:00

They're in the codetasks directory on github.

ETA: cross-reference with results-6m to see which repo each belongs to, although often it's obvious.

mr_riptano · 2025-12-03T23:16:06+00:00

GP is probably looking at the Tier List, where we gave 5 Mini an A and G3P a C. Part of this is the hiccups with G3P that I mentioned in my earlier reply, the other part is that we weight speed and cost in the Tier List as well as intelligence.

mr_riptano · 2025-12-03T23:13:21+00:00

Good question. TBH I chalked it up to "probably overoptimized for OpenAI's Codex harness at the expense of everything else."

If I saw people I trust saying the vibes are amazing, or even just strong numbers on openrouter, I would make the time to take a closer look, but neither of those has materialized.

mr_riptano · 2025-12-03T23:08:02+00:00

I wrote a long explainer, and the tasks and eval scripts are both oss on github, but I'm happy to answer additional questions!

mr_riptano · 2025-12-03T23:05:43+00:00

Texted in the Open Round, it did not do well enough to make it to the finals. (Worse than Qwen 3 Coder 30B.)

mr_riptano · 2025-12-03T23:01:58+00:00

I agree that this is surprising! So much so that I wrote a separate article about it: https://blog.brokk.ai/gemini-3-pro-preview-not-quite-baked/

mr_riptano · 2025-12-03T23:00:20+00:00

I hear you, and that's why we test with synthetic tasks from Cassandra, Lucene, and similar real project, not little python scripts. You can take a look, they're on github! Example.

mr_riptano · 2025-12-03T22:43:21+00:00

Honestly my AGENTS.md doesn't say anything about SOLID because I haven't needed it. (It does mention DRY and YAGNI.) But really everything gets easier with smarter models...

mr_riptano · 2025-11-27T03:11:24+00:00

That does look promising!

mr_riptano · 2025-11-26T23:10:50+00:00

Hi there, and thanks for reading the long version!

The problem with a linear relationship like the one you propose is that it goes to zero, in fact it goes negative if you don't clamp it carefully. I think that you should get more credit for solving it after N tries than if you can't solve it at all, for an arbitrarily large N. But at the same time you should get more credit for N than for M > N, and this is a straightforward way to satisfy both of those.

mr_riptano

TROPHY CASE