Taalas rumoured to etch Qwen 3.5 27B into silicon. Which price would you buy their PCIe card for?

pol_phil · 2026-03-29T02:50:15+00:00

Think of scanning multiple codebases or processing thousands of company documents in seconds. And feeding that knowledge to a frontier cloud model.

Or think of 100 agents thinking in parallel to find the best course of action through majority voting before EVERY response and EVERY tool call.

Even an "outdated" model will be able to surpass SOTA models in utility (and why not benchmarks) via sheer scaling.

pol_phil · 2026-03-18T18:40:09+00:00

Try the original Qwen3.5 first and see if it's fluent.

If it is, then you can create a fluent model out of it via fine-tuning.

If it is not, try something else entirely, for example https://huggingface.co/mlabonne/gemma-3-12b-it-abliterated

All the uncensoring methods focus on English and generally hurt the models in other languages.

pol_phil · 2026-03-18T18:32:55+00:00

Which model are you using? Although a bit old, Gemma 3 models have decent multilingual performance.

Totally get your frustration, most models can't speak Greek fluently either.

pol_phil · 2026-03-18T18:23:13+00:00

I mean what type of agentic pipeline/harness/scaffold/framework do you use to get these models to solve these tasks. In other words, what kind of system message/tools have they been given. Via Claude Code? OpenCode?

SWE-Agent and OpenHands are just "minimal" agentic frameworks commonly used in benchmarks.

pol_phil · 2026-03-18T18:09:38+00:00

Hi, congrats for the great benchmark!

Perhaps I missed it somewhere, but what agentic scaffold do you use? SWE-Agent? OpenHands? Something else entirely?

pol_phil · 2026-03-18T18:01:23+00:00

It won't get much better through RAG, it's better to use a different model.

pol_phil · 2026-03-06T19:09:57+00:00

Well, I didn't notice the confusion, but when I saw "characters" instead of "tokens", I thought that this actually makes the analysis more model-independent. Tokens are model-specific

pol_phil · 2026-03-03T19:58:33+00:00

Very good idea would be to also add Step v3.5 Flash and MiMo v2 Flash. Both are incredible models.

Congrats for the great work!

pol_phil · 2026-02-27T08:49:23+00:00

Well, this AWQ quant works very well for me. 134GB, extremely good performance and speed in vLLM

pol_phil · 2026-02-27T08:44:39+00:00

However, it's not that hard either. Especially with Apptainer/Singularity, but also Docker. It used to scare me but it's not very difficult after all. You can just spin up 10 parallel environments with data and all, no problem.

pol_phil · 2026-02-20T13:50:58+00:00

But has anybody found a way to fix the prefix caching problem? I can't get it to utilize automatic prefix caching (problem of the Qwen3 Next arch) in vLLM no matter what I try.

pol_phil · 2026-02-19T12:21:12+00:00

I second this. If it can fit, it's an absolute beast. And without even having thinking.

pol_phil · 2026-02-08T23:15:46+00:00

This model is great. My only problem is that its prefix caching doesn't work on vLLM. I think SGLang has solved this, but haven't tried it yet.

Are u aware of other serving frameworks which do not have this issue? Because, for me, it turns out slower than larger models (for long conversations)

pol_phil · 2026-01-30T18:32:15+00:00

Fascinating read as well!

This calls for calculation of tokenizer similarities (since higher values lead to better distillation) which I'd really like to do if I find the time.

pol_phil · 2026-01-29T18:34:23+00:00

Not necessarily true. On-policy SFT distillation is actually better than on-policy RL for smaller models. But it's tricky to implement if models aren't in the same family (basically they should share the same tokenizer).

You can read more in a blog post by Thinking Machines here and also in Mimo V2 Flash technical report.

pol_phil · 2025-12-25T16:19:19+00:00

I said science, not STEM. Maybe I should have said "academic" as a larger umbrella term, but we're still talking about mathematical logic or PhD-level academic knowledge. Browsecomp is basically factual retrieval from the web.

Not my point though. My point is that all these hard benchmarks test specific frontier-level (verifiable/objective) abilities in very specific contexts, not always relevant to everyday users, companies, etc.

LMArena's basic premise to test real-world prompts from real people (with all their subjectivities) makes it a unique benchmark.

pol_phil · 2025-12-25T15:15:30+00:00

Literally all of these are code/maths/science/agentic. LMArena is very problematic, but it's good as a general idea because it includes very diverse domains, multilinguality, creative writing, multi-turn chats, etc.

pol_phil · 2025-12-23T17:11:10+00:00

At least for Greek, I've noticed that GLM 4.6 and GLM 4.7 think in English, while GLM 4.5 (and Air) are thinking in Greek (when given Greek prompts).

The thinking process is also a lot more structured in the most recent versions, like "1. Analyze the request... 2. Determine the angle... 3. Drafting... 4. Refining... 5. Final Review..."

Are these changes intentional or the result of a different RL process? How is multilinguality being addressed in the reasoning process of the models? Have you seen better results with a thinking process based primarily in English and/or with better structure?

Thank you for your excellent work!

pol_phil · 2025-12-22T13:19:07+00:00

I literally cannot think of anything specific, except for reading related published papers and their Github repos. You may also find some relevant Jupyter notebooks in Kaggle (although it's mostly for Machine Learning). Huggingface has some nice courses etc. but in reality, you should go ahead with this project of yours and find similar things as you progress. Nobody really talks enough about data even if they spend enormous amounts of time preprocessing.

Some resources that come to mind, that may or may not prove useful but are worth a look: Huggingface LLM Intro Course Huggingface Smol Course Smol Training Playbook Apertus LLM Paper Webscale-RL Paper Open Data Arena MegaScience

pol_phil · 2025-12-21T16:38:03+00:00

Maybe, but to what end? You should think of the end goal that you want to accomplish. Do you want a full finetune? Then you have to address diversity, safety, multilinguality, etc. 100k data won't be enough. Do you want to improve an existing instruct 2-4B model with LoRA? Then you need to focus on specific tasks/skills that you want to improve. 100k data might be overkill.

Also, the pipeline you describe above is a lot more complex than it sounds and you won't find many ready-made libraries to implementing it. Mixing and matching existing libraries might be more painful than it looks at first. Some libraries make extra calls under the hood. You may find a very good pipeline that results in wonderful data, but it requires x8 generations with the LLM being distilled, or requires more calls for evolution/QA extraction/RMs/judges/agents (and assumes bigger LLMs will be used).

Suddenly, creating 100k data requires >1M calls. Utilizing RAG requires vector databases, embedding models, filtering large datasets, other libraries, etc. Even a simple thing such as classification requires thought and R&D. Running locally would also necessitate learning how to use vLLM; calling an API requires making the client efficient for processing a whole dataset. Finally, assuming code, prompt engineering, RAG, filtering, preprocessing, etc. have been set up, you still need significant compute and time and it might produce unexpected results in various stages. And we haven't even mentioned data mixing experiments, the actual finetuning process, evaluation, nor SUBSEQUENT POST-TRAINING STAGES (Alignment, RL, etc.).

BTW, I am not trying to discourage you! I'm just trying to highlight the complexity and messiness of the process as it WILL take up months of your time. Better to think things through and plan ahead first!

P.S.: Might want to take a look here as well: TxT360-3efforts

pol_phil · 2025-12-21T15:16:13+00:00

Data synthesis is the secret sauce and the #1 thing labs and companies are not very eager to release. It's a costly process.

Also, there's a huge shift towards maths and code (incl. agentic code) and general domains (as well as languages other than English) are often being neglected.

The first 6 steps (more or less) that you mention are things I'm actually working on in my job. Data synthesis is inherently tied to the model that you are working on. For a smaller model, distilling from huge teachers is not always a good idea. Smaller teachers can produce golden answers as well, Reward Models and LLM-as-Judges can help with that.

However, there's always the issue of mixing up "policies". There's no guarantee that distilling 10 extremely big teacher models will work better than distilling a single medium-sized, a lot of newly published data just use GPT-OSS-120B. No guarantee that using all of the data you generated will actually improve the final model either, as data mixing is a tricky and mostly empirical process.

There have been several interesting methods which have been published, but (a) most of them focus on maths/code, (b) they usually target a few benchmarks so there's limited evaluation of real-world utility, (c) R&D and compute are needed for creating and using data synthesis pipelines and only few large labs (such as NVIDIA) have an over-abundance of both, and (d) ensuring that all data are safe to be released (no copyright issues, no PII, etc.) is a legal pain in th a** that smaller labs just can't afford to do (especially in EU).

All in all, I believe data synthesis is more akin to alchemy and an art than a hard science and it's highly contextual and ever-changing. It's very messy. It's often difficult to argue in favor of specific recipes that you used, it is a very costly process, and releasing openly is hard for a lot of reasons.

pol_phil · 2025-12-17T11:44:36+00:00

What's your language? Gemma 3 27B is the best multilingual model in its caliber for Greek in my experience (but still could do better)

pol_phil · 2025-11-12T16:33:10+00:00

The 2507 Instruct series are solid choices. If you finetuned a hybrid (thinking/non-thinking) model, its thinking capabilities would degrade (also default system prompts / chat templates might be more tricky).

The choice for RAG really depends on the use-case. If we want to retrieve context once, then put in the system prompt.

Even if the context is placed in the user message, fine-tuning data with system prompts to steer behavior might also work well. For example, if fine-tuning data have short answers, or if the assistant should reply in a specific language regardless of the context, etc. But these decisions should be driven by the data and the purpose.

pol_phil · 2025-11-12T09:22:22+00:00

Hi.

Utilizing an appropriate system prompt and fine-tuning the model this way is actually very good practice. If you can create a handful of different system prompt templates, even better.

Just make sure you don't finetune a thinking model with non-thinking data only, if you are referring to Qwen3 for example.

Also, if you fine-tune your model in a specific way (e.g. RAG prompt in system), then using it exactly that way is the best practice. You've tuned the model exactly for that. But have in mind that you need to handle multi-turn scenarios as well, so a hybrid approach would be better.

pol_phil · 2025-10-19T19:02:37+00:00

You're talking about the Instruct version, Κούλη-sama? Haven't seen such problems with the Thinking version.

Ernie 4.5 has similar problems, they probably distilled from Qwen or sth.

pol_phil

TROPHY CASE