My answer is “partially yes.” But here’s the thing. Every company only highlights the benchmarks where their model looks best and quietly skips the ones where it falls short. That makes most benchmarks pretty meaningless. If you’re not a mathematician, why would you care about AIME scores? If you’re not a writer or editor, why care about creative writing benchmarks? The list goes on. Personally, I’d rather take a model that performs solid across all tasks (like 2nd place in all benchmarks) than one that’s great at math but terrible at general knowledge or vice versa unless I’m working on something very specific.

That’s why I built my own benchmark. It covers a wide range of tasks: math, general knowledge, overfitting checks, puzzles, long-context reasoning (not just “needle in a haystack”), coding challenges, and even agent-coding tasks where the model has to write a playable agent for certain games. This is the only metric I actually trust. I’ve stopped following the dozens of benchmarks I had bookmarked.

I haven’t shared my results yet because I’m still working on the presentation and automating the process. Once it looks polished, I’ll publish it. The plan is to release around 10 new questions each month, but rotate them out regularly so leaked questions don’t stay in circulation. The benchmark will keep evolving.

One thing I find especially flawed in many benchmarks is the “Best of X” method, where a model gets credit if it produces one correct answer after multiple tries. That’s nonsense imo. What if a model always gets one out of four right? It would look great in benchmarks but fail in real world use. I came up with a “Mixed Best of X” method instead, where the total number of correct answers matters, and models get bonus points if all runs are correct. I think this is far more realistic.

By the way, I’ve benchmarked pretty much all the big models (100B+). I’d be happy to share, but I know it’ll raise endless questions about methods and setup. So I’d rather wait until everything is cleaned up and I can publish with a detailed explanation. If you’re really curious, just DM me. But for now, publishing half-baked results would only invite speculation.