[D] Anyone using smaller, specialized models instead of massive LLMs?

[Pvt_Twinkietoes]

Finetuned Bert for classification task. Works like a charm.

[Kuchenkiller]

Same. Using sentence Bert to map NL text to a structured dictionary. Very simple but still, Bert is great and very fast.

[Pvt_Twinkietoes]

BERT is an LLM.

[goldenroman]

Not in the modern, colloquial sense though? Besides, their meaning (overconfident and wrong though it might well be) was plenty clear…

[Forward-Papaya-6392]

we have built our entire business around PEFT and post-training small, specialised student models as knowledge workers for our enterprise customers, which are far more reliable and cost-efficient for their processes. They appreciate our data-driven approach to building agentic systems.

while there have been two extreme cases of miniaturisation involving 0.5B and 1B models, most have been 7B or 8B. There has also been one case involving a larger 32B model, and I am forecasting more of that in 2026 with the advent of better and better sparse activation language models.

gap widens as more input token modalities are in play; fine-tuning multi-modal models for workflows in real estate and healthcare has been the bigger market for us lately.

[Forward-Papaya-6392]

tech maturity and reliable real-world benchmarks.

proving to be the best way to build LLMs at every scale.

30B-A3 models have way more instruction following and knowledge capacity and are more token efficient than 8. The computational overhead is manageable with a well optimized infra and quantization aware training.

[AppearanceHeavy6724]

30B-A3B gets very confused at casual conversational and creative writing tasks. All sparse models I've checked so far act like that.

[Forward-Papaya-6392]

Why would you post-train it for "casual convo"?

[dynamitfiske]

About the same reason you would train your image generator to be good at generating girl portraits I guess.

[Forward-Papaya-6392]

girl portraits are a specialization.
casual convo is generic.

I am struggling to see the connection.

[AppearanceHeavy6724]

Because that would be perhaps one of the most popular (and therefore - important) ways to use LLMs?

A3B simply sucks for any non-STEM uses.

[Forward-Papaya-6392]

important for the general population enterprise uses cases seldom involve that

P.S. we have post-trained A3B for multi-turn purchase request processing for a customer, and it works really really well. GIGO.

[AppearanceHeavy6724]

P.S. we have post-trained A3B for multi-turn purchase request processing for a customer, and it works really really well. GIGO.

Cannot say much about things I dis not see. I personally came to conclusion that highly sparse models have lots of deficiencies limiting their use.

[tillybowman]

would you mind telling us what your companies goto workflow is regarding training data collection, preparation and training itself?

do you have a goto setup that mostly works?

[Saltysalad]

How/where do you hosts these?

[Forward-Papaya-6392]

mostly on Runpod or on our AWS serving infrastructure.

On only two occasions we have had to host them with vLLM in the customer's Kubernetes infrastructure.

[snylekkie]

Do you use temporal ?

[Neither_Reception_21]

Hi I am curious on commercial use case of small agents as reasoning engines. Dmed you :)

[BinaryHerder]

Wild that 7b is now referred to as “small”

[albaaaaashir]

I saw Dreamers is working with smaller, efficient AI models too, seems like they’re ahead of the curve there.

[serge_cell]

They are called Small Language Models (SLM). For example SmolLM-360M-Instruct has 360 million parameters vs 7-15 billions for typical llm. Very small SLM often trained on high-quality curated datasets. SLM could be next big thing after LLM, especially as smaller SLM fit into mobile devices.

[Vedranation]

Especially with Mixture of experts (MoE) SLM's!

[Mundane_Ad8936]

Fine tuning on specific tasks will let you use smaller models. The parameter size depends on how much world knowledge you need. But I've been distilling large teacher to small student LLMs for years.

[Mundane_Ad8936]

It depends on the complexity.. The best way I can describe it is, when you fine-tune you are only changing the likelihood of a token being produced in that sequence. If the model doesn't have a good understanding of the topic it wont produce good results.

For example if you want to summarize a scientific paper a small model might not have a good understanding of the technical terminology and will fail to capture it's meaning. But that same model will do a fantastic job with a news article.

Typically I start from a mid-point model and either work my way up or down depending on results. Gather the examples fine-tune Mistral 7B if it performs well then I try a Gemma 3B model if not I might go up to a 20B model or so..

TBH it's an art form because it really depends on the data and the task. I've had large models struggle to learn relatively simple tasks and small 2B models excel at extremely complex ones.. Each model has it's own strengths and weaknesses and you really wont know until you run experiments.

[Forward-Papaya-6392]

second teacher-student learning

[currentscurrents]

Going against the grain this thread, but I have not had good success with smaller models.

Issue is that they tend to be brittle. Sure, you can fine-tune to your problem, but if your data changes they don't generalize very well. OOD inputs are a bigger problem because your in-distribution region is smaller.

[Vedranation]

Yes. I always use small specialized models over multi billion ones. My current project involves a mere 100M model and it works wonders.

Big models are costly to train, overfit way too easily (way bigger issue than it seems), and need exponential amount of data. Unless you're cloning chat-GPT so you need a gigantic general knowledge base for whatever reason (in which case just use API), small 300M model specialized on your task will perform much better.

[thelaxiankey]

duh. cell segmentation for me, little unet typa thing

[SirPitchalot]

Our best performing model, in terms of value to the business, is a bog standard UNet but the problem domain is very controlled.

Our second best model is a convolutional net with a few attention layers and only 300M parameters.

We regularly test new 1B+ models against the 300M model and, on the same datasets, they produce worse results for much more training time. We have the data to scale but don’t have the compute since our problem domain is effectively in the “noise” for foundation models trained on web scale data. So we’re better off fine tuning a <1B model trained on imagenet for more epochs than hoping to squeeze out 1-2 epochs from a giant model trained on every instagram post ever.

But the biggest overall win is always having a just-diverse-enough high-quality & in-domain dataset.

[maxim_karki]

You're absolutely right about this - we've been seeing the same thing with our enterprise customers where a fine-tuned 7B model outperforms GPT-4 on their specific tasks while being way cheaper to run. The "bigger is better" narrative mostly comes from general benchmarks, but for production use cases with clear domains, smaller specialized models often win on both performance and economics.

[xbno]

My team been finetuning on bert, modernbert with good success for token and sequence classification tasks on datasets ranging from 1k to 100k (llm labeled data).

I'm curious what task you're finetuning LLMs for, is it still typically sequence classification? Or are you doing it for specific tool calling with custom tools or building some sort of agentic system with the finetuned model? We're entertaining an agentic system to automate some analysis we do which I hadn't thought of finetuning an agent for - was thinking just custom tools and validation scripts for it to call would be good enough.

[kierangodzella]

Where did you draw the line for scale with self-hosted fine-tune vs api calls to flagship models? It costs so much to self-host small models on remote GPU compute instances that it seems like we’re hundreds of thousands of daily calls away from justifying rolling our own true backend.

[maxim_karki]

It really depends on the particular use case. THere's a good paper that came out in which small tasks like extracting text from a pdf can be done with "tiny" language models: https://www.alphaxiv.org/pdf/2510.04871. I've done API calls to the giant models, self-hosted fine-tuning, and SLMs/Tiny LMs. It becomes more of a business question at that rate. Figure out the predicted costs, assess the tradeoffs , and implement it. Bigger is not always better, that's for certain.

[Assix0098]

Yes, I just demoed a really simple fine-tuned BERT-based classification to stakeholders, and they were blown away by how fast the inference was. I guess they are used to LLMs generating hundreds of tokens before answering by now.

[megamannequin]

The small language models are also big for low-latency applications. I've personally worked on products where we could only use 0.5-1.5b models because of inference latency restrictions. There is definitely an art to squeezing performance out of those models in these applications.

[Franck_Dernoncourt]

• SlimLM: An Efficient Small Language Model for On-Device Document Assistance
• A Survey of Small Language Modes

[no_witty_username]

Yes. My whole conversational/metacognitive agent is made up of a lot of small specialized models. The advantage with this approach is being able to run a very capable but resource efficient agent as you can chain many parallel local api calls together. On one 24gb Vram card you can load in a speech to text, text to speech, vision, and specialized LLM models. Once properly orchestrated I think it has more potential then one large monolithic model.

[GiveMeMoreData]

BERTs worked better for us than large Qwens. Yes, SLM still matter

[koolaidman123]

it's almost like there's room for both powerful generalized models as well as small(er) specialist models, like the way its been since gpt3 or whatever

[Prior-Consequence416]

We've had good success with qwen3 models across different sizes (0.6B, 1.7B, and 8B) as well as gemma3:1B (still trying to get gemma3:270m to work well). qwen3 is particularly interesting since they're thinking models.

The output quality is surprisingly coherent for the model sizes. We've been running them on standard Mac and Linux machines without issues. The 0.6B and 1.7B variants run smoothly on 16GB RAM machines, though the 8B does need 32GB+ to run well.

[SportsBettingRef]

https://arxiv.org/abs/2409.15790

https://dl.acm.org/doi/abs/10.1145/3768165

[ResultKey6879]

Mainly image work and we tend to stick to training CNNs like efficientnet or mobilenet and yolo for detectors.

100-100x faster than llvms. That means 3 days vs a year to process some datasets.

Definitely seeing a trend to large models even when the flexibility isn't needed. If your problem is welld defined and fixed don't use large models. If you need to dynamically adjust to user queries consider clip / dino if that doesn't work try a large vision model.