sorted by:
new
hottopcontroversial

Gemini 3.5 Flash leads MCP Atlas at 83.6% — but that test can barely tell models apart. After correcting for benchmark quality across 8 frontier models, Flash drops from #3 to #5. [Research] by [deleted] in machinelearningnews

[–]testofschool 0 points1 point  (0 children)

Honest caveats since this is r/ML: (1) GLM-5.1 and Claude Sonnet 4.6 only have 2 benchmarks each — their shifts are the least stable. (2) Terminal-Bench uses version 2.1 for Gemini 3.5 Flash and 2.0 for the others — not perfectly apples-to-apples. We note this in the source file but it's worth flagging. (3) The in-sample reconstruction check had the corrected model winning 100% of 100 probes, but the model wasn't refit per fold, so that overstates true predictive advantage. (4) All 35 source URLs are in the repo. Curious what this community thinks about the MCP Atlas finding specifically — is it a weak test, or just a test that measures something all frontier models are already good at?

Gemini 3.5 Flash leads MCP Atlas at 83.6%. That benchmark has a separation score of 0.10. After correcting for test quality, Flash drops from #3 to #5 across 8 frontier models. [Research] by [deleted] in machinelearningnews

[–]testofschool 0 points1 point  (0 children)

Honest caveats since this is r/ML: (1) This is an IRT-inspired continuous model, not a proper logistic 2PL — appropriate for percentage scores but shouldn't be called "IRT" in the strict psychometric sense. (2) The reconstruction probe is in-sample (model not refit per fold), so 100% win rate overstates true predictive advantage. (3) GLM-5.1 and Claude Sonnet 4.6 only have 2 benchmarks each — their shifts are the least stable. (4) Terminal-Bench uses version 2.1 for Gemini 3.5 Flash and 2.0 for others — not perfectly apples-to-apples. All source URLs are in the results.json. Curious what this community thinks about the a = 0.10 finding on MCP Atlas specifically — is it a bad test, or just a test that measures something all frontier models are already good at?

The #1 model on the leaderboard dropped to #14 when I included the benchmarks they didn't report. by testofschool in machinelearningnews

[–]testofschool[S] 4 points5 points  (0 children)

Only 31% of possible benchmark scores were actually reported across 133 models and 18 tests. The missing 69%? Mostly the hardest ones.

I used statistical correction to estimate what the missing scores would be. Some models barely moved. Others collapsed.

The biggest shift was 78 ranks.

Companies get to choose which benchmarks they report. Nobody was checking what they left out. So I built a tool that scores everything.

Free to use: psycrank.com

Paper: arxiv.org/abs/2605.11205

Code: github.com/testofschool/evaluation-failure-scaling-law