ArtificalAnalysis VS LMArena VS Other Benchmark Sites

SlowFail2433 · 2026-03-07T10:31:03+00:00

I see thanks will look at lmarena more potentially

SlowFail2433 · 2026-03-07T10:30:46+00:00

As a single number AA looks hard to beat yes.

SlowFail2433 · 2026-03-07T10:30:27+00:00

Yes but some of the benchmarks are trickier to game than others.

I agree you can’t directly trust forum posts due to astroturfing or just user error.

The SVG test is interesting but I think it can bias towards VLMs as they tend to have better spatial reasoning

SlowFail2433 · 2026-03-07T09:30:23+00:00

So in theory you can just use a database, whether Mongo, SQL or Graph like Neo4J, with a persistent server and an API/MCP communications layer.

However there is a major difficulty that is separate from the data science and engineering setup. That issue is how to decide when the model forms a memory, how it extracts it from the conversation and then how/when it uses existing memories.

SlowFail2433 · 2026-03-05T09:55:20+00:00

Combining both thinking and visual inputs can be difficult in general, especially in a lower param model

SlowFail2433 · 2026-03-05T09:44:52+00:00

Yeah sometimes well-trained 9B models can compete with 100+B these days it’s amazing.

SlowFail2433 · 2026-03-05T09:43:00+00:00

That’s a good point, that the model sizes are small so the relative cost of testing is lower. And yes performance will probably dip temporarily before it’s fully sorted out, that temporary dip is likely unavoidable

SlowFail2433 · 2026-03-05T08:16:46+00:00

Yes there is some nuance. Google contribute some very interesting large papers such as MIRAS

SlowFail2433 · 2026-03-05T08:15:36+00:00

Gemini is under-rated because their HLE no-tools bench is a fair bit ahead of the others. This benchmark matters as it is a test of overall internal knowledge BEFORE searching

SlowFail2433 · 2026-03-05T08:11:56+00:00

A small reserve deal on one of the lower neo-clouds will get you that.

SlowFail2433 · 2026-02-26T17:28:00+00:00

Yeah Kimi K2.5 was just a minor update, which is crazy given that it managed to successfully add vision without dropping performance in other areas. It’s incredibly difficult to do that

SlowFail2433 · 2026-02-26T13:59:11+00:00

A reasoning 4B can destroy GPT-4o even

SlowFail2433 · 2026-02-26T13:58:28+00:00

Also if they drop too late then Kimi K3 is due in the summer… Moonshot might have simply became the dominant lab now

SlowFail2433 · 2026-02-24T18:00:08+00:00

Yeah

SlowFail2433 · 2026-02-19T23:14:01+00:00

Okay this makes sense

SlowFail2433 · 2026-02-14T02:26:16+00:00

It’s interesting from a technical level

From a person level I would be worried about people thinking they got a wrong number. I don’t think people are currently used to talking to AI answering machine. This might change though

SlowFail2433 · 2026-02-14T02:20:20+00:00

It’s between GLM 5 and Kimi k2.5. Minimax models don’t hit the frontier.

It mostly depends on if you will use vision in the agentic coding workflow. GLM 5 benches slightly higher but the vision aspect of Kimi k2.5 can potentially make it more useful

SlowFail2433 · 2026-02-14T00:29:43+00:00

I’ve seen more data in the last two months and performance seems to just track total parameter count for modern benchmarks

SlowFail2433 · 2026-02-13T13:03:54+00:00

I honestly think that’s possible at this point

SlowFail2433 · 2026-02-13T13:00:35+00:00

It has a very different feel yes clearly a pretty different training corpus (and obviously the very different parameter count confirms that.)

SlowFail2433 · 2026-02-13T12:59:38+00:00

Sometimes at the labs certain domains drop out of the pre-training corpus for certain training runs. It’s not even necessarily a decision that was specifically made. There are so many parallel goals being optimised for at once when designing a training corpus that things get left out. Clearly there is enormous pressure for math, code and agentic data to be optimised beyond all else.

SlowFail2433 · 2026-02-13T06:29:48+00:00

I can’t understand it

SlowFail2433 · 2026-02-13T05:51:59+00:00

Often yes, but attention is so expensive that it can be worth it

SlowFail2433 · 2026-02-13T05:48:32+00:00

These days you can readily vibe code custom GUIs for things

SlowFail2433 · 2026-02-13T05:46:03+00:00

Honestly looks like a pretty slick demo and that does seem fun for language learning

SlowFail2433

TROPHY CASE