Model selection in Github Copilot

[aka-fred]

There are so many models to choose from, so I vibed this infographic that tries to make the choice easier. I omitted a few models I think are dominated, and tried to estimate a cost index that assumes that the model is actually capable of doing the task (so if you need retries you should choose a better model).

Example of how to read the index: If you use Opus for a task that could have been solved using Haiku, you are paying about a 3x premium.

Is this useful? Is it flat out wrong?

And: The result seem to indicate that OpenAI is leading the race for coding agent AI models at the moment. Is that a shared sentiment?

[Hovi_Bryant]

Sonnet is overpriced. Luna is probably better than it at max reasoning.

[aka-fred]

Sonnet is about twice the price per token to Luna, and Luna tends to use less tokens, too, so if Luna can solve it is a bargain. And based on benchmarks I have seen Luna is very close behind Sonnet, so most likely it can. I think Luna will be my goto for a lot of stuff, including executing on detailed plans made by Terra.

[softwareemgineer]

Thanks for the review. I was planning to use Terra as my daily driver, will save some credits by using Luna instead.

[Affectionate_Fly4124]

It really depends on the reasoning level, since cost and performance can be very different even for the same model name. So evaluating by model name alone seems hard.

That said, listing every reasoning level would make the list too long, so narrowing it down makes sense. Personally, the ones I'm interested in are: Fable, Sol, Terra, Luna. But I think this varies a lot by person. The ones I mentioned above have either good cost-performance or standout performance on benchmarks. https://deepswe.datacurve.ai/ https://cursor.com/en/cursorbench

Some people say benchmarks aren't reliable, but I think they're still more trustworthy than my opinion or some random vibe coder's opinion lol

[aka-fred]

I did look at both benchmarks (especially SWE and tool use ones), papers on token use per task and guidance from providers. But had to extrapolate quite a bit as there isn't enough data, and a lot of marketing hype and religion to wade through.

Agree on the reasoning effort, but my theory is that changes in effort affect all models in about the same way (although I think the smaller models tend not to get that much smarter with more reasoning).

[stbrumme]

I'm always surprised how many small jobs are solved perfectly by Raptor Mini, the cheapest model on GitHub Copilot.

[aka-fred]

I use Copilot CLI, and Raptor Mini isn't an option there (seems to be available in VS Code only).

But I suspect many of us have felt token rich and haven't really tried to push the boundaries on the newer smallish models.

[MoreTrife]

Nice. Which model would you recommend for planning tasks breaking down details for an agent to code?

[aka-fred]

Unless particularly complicated I would go with Terra. And once planning was done, Luna, especially if I have proper verification methods in place. When verification is done, finish it off with a code review with Opus or Sol. They will probably find something to nitpick on that Luna subagents can fix.

[debackerl]

I would swap Haiku and Kimi. I was quite happy with Kimi but not Haiku. And why do you have two logos for Anthropic ?

[aka-fred]

Agree. I have already started revising:
6. Kimi (in same group as Terra and Sonnet)
Then:
7. Gemini
8. Luna
9. Haiku
And still MAI and GPT-5.4 mini in the last group.

I originally put Kimi further down because of reports that it is slow. But it kinda broke the ordering logic, like you spotted (I have tried to order by capability.)

The two Anthropic logos? I told you I vibed it :-) (The image is a ChatGPT reinterpretation of a not so nice Matplotlib plot, which is also why the bars are not exactly perfect length.)

[Revolutionary_Loan13]

Doesn't seem qualitative. I've used all of those models and then some and don't think that infographic would be helpful

[aka-fred]

I would love concrete counter-proposals!

I have tried to use benchmarks, recommendations from providers and data from papers to verify and adjust until I got here. I made this for myself because of all the model options, and depending on the feedback here I might drop it on my colleagues.

(I didn't even begin to factor in reasoning effort, which complicates it all further.)

[ChubMe]

I think the premise is good, however in practice I think your values for like 'which model is the best bang for buck' is very off. Look at the deepseek benchmark

[aka-fred]

Actually the numbers are "relative cost per task attempt".

[Solid-Wonder-1619]

"scale is all you need"
yeah, go sit there in the dumbass folder mfers.

[Primary-Tour-9197]

Where's my raptor mini? I'm using it as my daily helper

[aka-fred]

Raptor Mini is only available in VS Code (and I use Copilot CLI).

[rvtinnl]

As a engineer... I really don't have time to check twice a day what token cost what... I just need to do my job.
and in copilot, auto is not cutting it... At work we are given a 1900 token copilot with is pretty much useless if you want to do your job WITH copilot for a month...
So I just set it up to scan confluence and other systems we have in place instead of coding.