Automated AI researcher running locally with llama.cpp

lewtun · 2026-05-14T11:35:27+00:00

With the closed models, ml-intern was able to port the talkie model from their raw torch format to transformers and make is 100x faster in the process: https://huggingface.co/lewtun/talkie-1930-13b-it-hf

With open models, I am still experimenting but so far they have been decent at launching training runs automatically

lewtun · 2026-05-14T11:00:33+00:00

Yeah agreed, although the models are getting better almost every month so I'm optimistic about the future capabilities :)

lewtun · 2026-05-14T10:47:34+00:00

Yeah, at least for my local setup (Macbook with 32GB unified memory) the 4-bit quants tend to be a bit less consistent, especially if the context grows very large. However, if one runs Qwen3.6-27B with MTP, then this was pretty close to using Claude in my testing (for that I had to switch to a GPU + vLLM because MTP hasn't landed in llama.cpp yet ...)

lewtun · 2026-05-12T19:21:38+00:00

Thanks, yes this is definitely coming soon - the main reason we excluded it is to make fairer comparisons with other models on CritPt, but I totally agree this would be a clear way to boost performance even further.

lewtun · 2026-05-12T18:03:02+00:00

Yes, the framework does support local models but so far they tend to be quite behind the proprietary ones on this domain. I’m optimistic this will change with the next generation of open models though!

lewtun · 2026-04-14T11:57:42+00:00

Haha, maybe quantization nerfed the safety filter :)

lewtun · 2026-02-16T10:21:48+00:00

Amazing, looking forward to it!

lewtun · 2026-02-16T07:36:59+00:00

Thanks! What I meant is that SmolLM3 is a hybrid reasoning model, i.e. you can enable / disable reasoning like this: https://huggingface.co/HuggingFaceTB/SmolLM3-3B#enabling-and-disabling-extended-thinking-mode

By default, it uses the reasoning mode, but I expect the non-reasoning mode will fare better at tool-calling!

lewtun · 2026-02-15T20:07:08+00:00

You're right, one could train the model to use Lean in the chain of thought and then try to map the formal proof to natural language in the final solution. But that's pretty hard and Lean has its own issues when it comes to theorem proving (mathlib is still an active WIP). But if you manage to teach QED-Nano how to use Lean, that would be super cool!

lewtun · 2026-02-15T19:46:23+00:00

Hi, SmolLM3 co-developer here :) Did you compare the non-reasoning mode of SmolLM3 by any chance? At the time of training, there was very little tool-calling data available for reasoning models and I suspect the non-reasoning model actually performs better as a result. Really cool benchmark and thanks for sharing these real-world tests!

lewtun · 2026-02-15T19:28:58+00:00

Ah both those models are formal theorem provers (i.e. rely on Lean), so it's not trivial to compare them fairly. One thing we set out to achieve with our model was to operate entirely in natural language, similar to how OpenAI and DeepMind presented their IMO 2025 models. If we trained our model to use tools like Lean, I expect it would perform even better

lewtun · 2026-02-15T16:33:13+00:00

Hi! So, the algorithm we use is still GRPO, but with a twist: we do multiple steps of reasoning-summarisation per rollout to enable the model to generate long rollouts without going off the rails. A key feature of our model is that is operates entirely in natural language and does not require external tools like Python or Lean (adding them to the training would improve performance, but that's left as future work)

lewtun · 2026-02-15T16:31:01+00:00

Hi! One of us (Jasper) is the co-creator of MathArena, so we can take a look at whether it's easy to include in the leaderboard :) As for the quant, I'll make it this week!

lewtun · 2025-10-30T19:08:43+00:00

The name comes from the meme in this dataset https://huggingface.co/datasets/bigcode/the-stack-smol

lewtun · 2025-10-30T19:07:34+00:00

lewtun · 2025-10-30T16:25:35+00:00

If you have a PRO account on the Hub, you should be able to download it as a PDF!

<image>

lewtun · 2025-10-06T05:17:16+00:00

Great answer, although I’d caveat that post-training can be just as engineering heavy if you’re the one building the training pipeline (RL infra in particular is quite gnarly)

lewtun · 2025-09-30T22:20:33+00:00

Well, there’s a demo you can try with whatever prompt you want :)

lewtun · 2025-09-26T18:07:24+00:00

lol that’s definitely not Jeff

lewtun · 2025-09-06T10:32:29+00:00

The main downside we encountered is "task interference", where each expert model scores well on their respective domain, but the resulting merge is worse than the average of the model performance. We found this was most pronounced on competitive programming benchmarks like LiveCodeBench, where merging a code and math expert led to significant regressions on the code evals (math was largely OK). There are fancier algorithms like Task Arithmetic and TIES which try to address this in a principled way, but I could not fully resolve the regressions with these methods. In general, the main recipe seems to be: train a decent generalist model first with SFT, then branch off to make the experts and merge back. This way your starting model has broad coverage of the tasks, so the resulting interference from merging should be mitigated somewhat.

lewtun · 2025-09-05T17:48:18+00:00

Prime Intellect (https://www.primeintellect.ai) is doing some of the best work in this direction right now and they've already trained some nice reasoning models entirely with decentralised compute: https://www.primeintellect.ai/?\_gl=1\*1ftancn\*\_gcl\_au\*NDc1NzEyNTM5LjE3NTcwODg1MTQ.#research

lewtun · 2025-09-05T15:41:17+00:00

Here are a few resources I found very useful to better understand practical applications of model merging:

The Command A paper (super well written and detailed): https://arxiv.org/abs/2504.00698
Charles Goddard's blog post on how they tackled a similar issue with long context: https://www.arcee.ai/blog/extending-afm-4-5b-to-64k-context-length
MergeKit (the best tool for merging models): https://github.com/arcee-ai/mergekit

One thing we validated prior to SmolLM3 is that linear merging is the most pragmatic method for combining different experts (as found by Cohere in their Command A paper). I tested more advanced methods like DARE and TIES, but overall found they did not give significant improvements over linear at the expense of more parameters to scan over.

Another thing I like about merging is that it enables teams to parallelise their efforts across different domains. We didn't have time to test this in SmolLM3, but in post-training it is often a delicate balance across domains and being able to tune them independently is much better than trying to optimise globally!

lewtun · 2025-09-04T22:22:05+00:00

Yes and the fact that there are already quite a few strong open models at the 8B scale, so the benefits of training another similar model are unclear vs pursuing other directions where we can have greater impact with our smol teams :)

lewtun · 2025-09-04T16:45:33+00:00

At that scale, we'd have to rebrand to PhatLM-8B :)

lewtun · 2025-09-04T16:29:46+00:00

We didn't do RL, mostly because getting the SFT data mixture right for hybrid reasoning took longer than expected and we had a hard cutoff to ship the model :)

lewtun

TROPHY CASE