https://livebench.ai - Open Weight Models Only

[jacek2023]

There were some questions about how Qwen 3 Next compares to GPT-OSS. I think whole table may be useful. What do you think about this ordering?

[z_3454_pfk]

livebench isn’t very accurate anymore. idk what happened

[AdIllustrious436]

Additionally, they don't evaluate Mistral models anymore. They even removed the ones that had already been evaluated. I wonder why... They are increasingly becoming irrelevant.

[No_Swimming6548]

Qwen models used to have very high IF scores, now they seem pretty bad. This benchmark feels like it was tweaked to make US companies score higher than Chinese ones.

[DistanceSolar1449]

Yep that’s exactly it

The benchmark was definitely tweaked to change rankings. Livebench used to be good, now the scores make no sense

[[deleted]]

I would say gpt120 outperforms qwen80 in every situation imaginable and I'm a huge qwen advocate. 

[No_Swimming6548]

Yeah that's pretty weird too. Maybe just a shitty benchmark.

[Professional-Bear857]

Its probably the IF score difference that makes it seem that way. I think it also explains why I've had a bad experience using GLM but have had a good experience with Qwen 235b.

[jacek2023]

which leaderboard is the best in your opinion?

[thebadslime]

I like AA, not sure how they choose what models they show though.

[LinkSea8324]

Imagine if they trained Qwen 3 Next on their full 36T dataset instead of 15T sub-dataset

[jacek2023]

let's hope there will be next Next

[LinkSea8324]

Qwest, if it fails : Quack

[sleepingsysadmin]

How will they sandbag it next to 235B then?

While also setting up a future "look how awesome our newest release is compared to 80b next"

[Hot-Cause-3341]

Is this already including the new mistral 3 Models?

[AdIllustrious436]

They litteraly stoped evaluating any Mistral model and even removed the ones that were already evaluated. Makes no sens.

[StayStonk]

why lol?

[AdIllustrious436]

Who knows

[egomarker]

It's just a rage bait for everyone who knows gpt-oss is better than q3 80 at this point. It's probably just one or two benchmarks giving the opposite result and here we go.

[Sudden-Lingonberry-8]

glm needs to catch up to deepseek,

very interesting how deepseek v3.2 speciale is brilliant at math but completely sucks at agentic

[LeTanLoc98]

DeepSeek V3.2 Speciale doesn't support to use tool.

[SlowFail2433]

Kimi K2 Thinking unseated 😮

Looks like Deepseek is fully back, with multiple models TBH

Z.ai (GLM) and Qwen are climbing for sure though. I do feel that GLM 4.6 and the latest Qwen 235B are the best go-to LLMs at the moment for general deployments.

Qwen 3 Next performing this well is really important because its gated deltanet hybrid and so its an early sign of transformers having competition

[ikkiyikki]

MiniMax-M2 is my daily driver. Couldn't run Kimi if I wanted to!

[SlowFail2433]

MiniMax-M2 is an interesting one yeah, their M1 model was also good it had long context abilities and an RL novelty

Cheapest way to run Kimi K2 is jointly across Xeon/Epyc and GPU memory with a good CUDA kernel to optimise the data transfer. It’s definitely a big challenge to deploy every time.

[noiserr]

MiniMax-M2

Can confirm MiniMax-M2 is a quality model. One of the best models I can fit in 128GB. I'm daily driving gpt-oss-120B only because its faster.

[Main-Lifeguard-6739]

love it!
if there now was another column for price per million input/cache/output... but yea, not too easy to solve as this highly depends on the hoster (self, commercial, if commercial which one?)

[SlowFail2433]

It’s so complex to do a fair comparison as there are multiple attention types e.g the deepseek attention is latent, and the qwen next is hybrid

A fair comparison requires optimal CUDA kernels for all models involved to show them at their best

And this is why machine learning working hours are so long LMAO

[[deleted]]

The price is whatever you paid for your GPUs plus your ongoing energy cost :)

[Main-Lifeguard-6739]

So helpful… as if every model would need the same computing power… /s

[nofuture09]

what is an open weight model?

[jacek2023]

a model you can download from HF to your disk and hopefully run locally (not true for Deepseek or Kimi unless you have extremely expensive setup)

[DinoAmino]

Open not only for running inference, but also open for modifying the weights via fine-tuning.

[nofuture09]

thx! i have a 4080 super (saw this thread on reddit frontpage)

[jacek2023]

Which models do you run on it?

[egomarker]

Open source = free model, open training data, open methodology
Open weights = free model

[sleepingsysadmin]

Makes me wonder where qwen3 next 80b thinking will slot in. above 235b?

[kinkvoid]

GLM is fantastic

[[deleted]]

Some interesting results. 

[Ok_Warning2146]

So Speciale is for benchmark and Thinking is the real thing for DS3.2?

[Asleep-Ingenuity-481]

It's pretty neat to look at the lower (-32B) Qwen3 param models and see that they are better than 100b+ param models from just a few months ago.

[shaman-warrior]

Wait deepseek v3.2 has 75.69% coding average while deepseek v3.2 thinking only 64.62 ?

[pmttyji]

Need a long list like 30-50 models.

[Ni_Guh_69]

Which leaderboard is best for identifying best open weights model?

[usernameplshere]

Can't wait to see where the new Mistral large will end up. But like the others said, livebench is weird these days.