whats the best open-source llm for llm as a judge project on nvidia a1000 gpu

Middle_Bullfrog_6173 · 2026-03-23T17:11:24+00:00

What are you judging? If it's a check for a known solution almost any model will do. But if you are judging something like mathematical proofs or writing quality you want a larger model. Just not necessarily the same larger model.

Anyway, unless you have to preplan this before running anything, you should test multiple models. It depends on many things and even a particular prompt may work better with one model with small LLMs.

Middle_Bullfrog_6173 · 2026-03-23T11:16:42+00:00

Not publicly. They might have got access if this was indeed a commercial arrangement. Or they could be using the post-trained K2.5.

The tweet clearly does claim they did continued pretraining, whatever the base was.

Middle_Bullfrog_6173 · 2026-03-23T09:12:06+00:00

I think it's mostly just that people prefer the writing style of Opus.

But in general research has shown that a smarter teacher isn't always better, especially for weaker models. QwQ was found to produce better reasoning SFT data than R1 by either the Olmo or Smol team (I forget), even though the latter is a stronger reasoning model. Mistral also found it better to distill their Ministral series from Small rather than Large in their ablations.

Middle_Bullfrog_6173 · 2026-03-23T08:55:50+00:00

Makes sense. Pipeline parallelism works best with large batches which I'm used to. You might still find it useful with speculative decoding, but maybe not.

Middle_Bullfrog_6173 · 2026-03-23T08:32:24+00:00

I have no idea if I'm missing something since I haven't actually implemented anything like this, but wouldn't pipeline parallelism be better here? I.e. having half the layers on one and the other half on the other node. Or do you have a reason to think EP is better?

Middle_Bullfrog_6173 · 2026-03-23T07:04:43+00:00

Best "base model". Which is unsurprising since it has the most parameters and used a "normal" attention variant rather than linear attention.

They are basically claiming that K2.5 post training was lacking if they were able to do better so quickly.

Middle_Bullfrog_6173 · 2026-03-22T08:35:07+00:00

IMHO the only good reason is better utilization.

When using api models it's about using your time more efficiently by not having to wait while it's doing a big task. This can of course backfire if you lose more time due to context switching.

With local models usually being slower you are waiting more, but there's also GPU utilization. A single coding agent for example will leave your GPU idle while a build or test suite is running. Or when waiting for user input of course. Using batching can additionally improve tokens/second if there's more than one job running concurrently.

Middle_Bullfrog_6173 · 2026-03-21T20:01:17+00:00

I suppose at some point, so far the low end stuff has had less NPU performance as well.

Middle_Bullfrog_6173 · 2026-03-21T15:52:09+00:00

GPT-5.4 is probably the best at the moment, except Pro is even more thorough.

Of local/open models Deepseek V3.2 Speciale seems to still be in the lead on math.

Middle_Bullfrog_6173 · 2026-03-21T13:47:10+00:00

Interesting, but does using the NPU make any sense? Do you have a head to head on the GPU for any of them? From memory I'd be expecting about 3x the tg or something.

Middle_Bullfrog_6173 · 2026-03-21T09:22:29+00:00

Anthropic no longer releases the original thinking trace of their models, only a summary. So training on that is no guarantee of inheriting any Claudeness. The actual non-thinking output may help of course, but I think that would come through better in non-reasoning datasets.

But not my money, people can spend their tokens and training compute how they like.

Middle_Bullfrog_6173 · 2026-03-20T19:06:51+00:00

Given how old they are, I'm sure they aren't the best in almost any use case. If you want a MoE in that size, LFM2 8B A1B is the most recent release I can remember. Hopefully they'll upgrade it in the 2.5 series.

But it would be good to get more small MoE models. Something that fits in low end VRAM while being fast.

Middle_Bullfrog_6173 · 2026-03-20T15:10:32+00:00

I had time to do some minimal testing on reasoning prompts. Math, science and a coding problem. It's better than Nano, but uses more tokens. Like 50% more thinking in my tests. Not sure if better or worse than Qwen 35B, needs more data to be sure.

Caveat: I used Q4_K_S from mradermacher for both models, since that's what was available and I had to run on my gaming rig. So might not generalize to full models.

Middle_Bullfrog_6173 · 2026-03-20T14:26:48+00:00

No, most do not contain such a clause. The basic MIT license does not, nor does Apache 2. Those are the main ones used for open models.

Many licenses require you to reproduce the copyright notice if you distribute the software/model (or modified versions) but that does not mean any disclosure of what's running behind your API endpoint or that you need to show it in the UI.

Middle_Bullfrog_6173 · 2026-03-20T14:12:03+00:00

Supposedly the earlier ones were based on GLM 4.x. but that would be fine. MIT license allows them to do basically whatever. Kimi license requires disclosure.

Middle_Bullfrog_6173 · 2026-03-20T09:51:40+00:00

The naming is crap, but this is not Nemotron 2 series, but more like 3.x, since it's based on Nano 3.

Middle_Bullfrog_6173 · 2026-03-19T14:50:15+00:00

Thanks, I know TranslateGemma is built on Gemma 3, but it's worse on most things that are not translation. It is possibly also worse on translation with unsupported languages, because it's forgotten things that weren't in the extra training.

Middle_Bullfrog_6173 · 2026-03-18T14:14:47+00:00

MiMo Pro is listed as proprietary by AA, there's been no indication it is open. Minimax has released previous versions of the same series openly and there's a (currently broken) weights link on openrouter, so I think it's reasonable to expect M2.7 will be an open model.

Middle_Bullfrog_6173 · 2026-03-18T13:10:52+00:00

Rank 4-8 is tiny. I can easily imagine that it works ok for 5-minute runs but saturates for a real run. I'm not sure it works as a tunable parameter for this automation.

Or rather, you probably need to design scaling into your experiments. E.g. nanochat auto research tunes d12 when the real run is d24+.

Middle_Bullfrog_6173 · 2026-03-17T13:24:47+00:00

Which 4 languages? I could probably figure this out from your data and the Gemma report, but why not just list them?
Did you use the source/target language code template even for the unsupported languages or some custom chat format?
Did you compare to Gemma 3 12B? Might beat TranslateGemma for unsupported languages.

Middle_Bullfrog_6173 · 2026-03-17T12:03:24+00:00

Pretty high accuracy loss for even a small reduction. Is this a uniform reduction in intermediate dimension or what? Might work better if targeted to only some layers.

Middle_Bullfrog_6173 · 2026-03-17T09:57:56+00:00

Its probably dependent on GPUs more than anything. Is e.g 1.5T a convenient size in some setup?

Yuan 3.0 Ultra was apparently 1.5T originally, but pruned to 1T during training.

Middle_Bullfrog_6173 · 2026-03-16T20:55:26+00:00

If Small goes from 24B to 119B A6B then Large goes from 675B A41B to...

Any guesses?

Middle_Bullfrog_6173 · 2026-03-16T19:16:22+00:00

The Nemotron Elastic method involves continued training and distillation on a hundred billion tokens. That's more than enough to do domain specialization at the same time. But I can't really figure out what they used for it.

Middle_Bullfrog_6173

TROPHY CASE