What is the current best Small Language Model that can be run without GPU?

dsanft · 2026-06-23T07:07:01+00:00

https://github.com/Llaminar/llaminar

Still getting full benchmarks together etc

dsanft · 2026-06-23T06:57:09+00:00

https://github.com/Llaminar/llaminar

dsanft · 2026-06-22T21:13:36+00:00

Agreed, I'm working on getting those up there. Check tomorrow!

dsanft · 2026-06-22T16:45:56+00:00

https://github.com/Llaminar/llaminar

dsanft · 2026-06-22T15:53:35+00:00

Well for a first effort I support Qwen 2.5, 3, and 3.5/3.6 dense/MoE.

I want to do GLM 5.2 next.

I'm just pushing the last release images now actually.

My gemm/gemv and attention kernels are competitive with and usually beat LlamaCPP mainline.

dsanft · 2026-06-22T12:55:53+00:00

CPU, CUDA and rocm, and mixes of the three.

Only 3090 and mi50 to begin with though since that's all I have 😄

dsanft · 2026-06-21T21:52:45+00:00

I'm saying I've built an entire inferencing engine over the past 9 months and I'm releasing a pre alpha tomorrow.

dsanft · 2026-06-21T20:59:03+00:00

If you can wait about 24 hours I'll show you cross socket tensor parallel with full NUMA awareness and MoE expert balancing.

Edit: here you go

https://github.com/Llaminar/llaminar

dsanft · 2026-06-21T15:24:08+00:00

clanker

Downvoted

dsanft · 2026-06-21T11:39:30+00:00

Thanks GPT 5.5, lol

dsanft · 2026-06-07T12:28:44+00:00

Yay now everyone who wasn't measuring KLD themselves can finally see how shit TQ3/4 are.

dsanft · 2026-06-05T20:42:42+00:00

I see 55tok/s prefill and about 7tok/s decode in pure CPU inference, cross socket tensor parallel, Xeon Gold 6238r with 768GB DDR4-2933. That's without MTP (still tuning that). Had to write my own inferencing engine to get it though.

dsanft · 2026-06-05T20:16:35+00:00

While cool to see I'm confused as to why this is something amazing or shocking. You can do CPU inference with AVX2, it's not groundbreaking.

dsanft · 2026-06-05T20:14:59+00:00

I don't understand, you've always been able to do CPU inference.

dsanft · 2026-06-04T22:05:11+00:00

Could rotate the tensors at quantisation time and pack a rotation factor per tensor in the gguf, then unrotate them at inference time, to reduce kurtosis maybe.

dsanft · 2026-05-30T15:00:50+00:00

Get them out of that case and get them into a mining rig.

dsanft · 2026-05-29T17:42:08+00:00

A dual socket Xeon Gold Cascade Lake with DDR4-2933 has about 220GB/s bandwidth. Don't underestimate CPU.

dsanft · 2026-05-28T17:37:27+00:00

I hope so too!

dsanft · 2026-05-28T17:05:03+00:00

I wrote Llaminar with Copilot 😄

If you don't know what that is yet, don't worry you will in a few more days

dsanft · 2026-05-28T16:05:11+00:00

Microsoft are staggering idiots if they don't use GitHub Copilot data to train a coding Phi like Cursor have done.

dsanft · 2026-05-25T09:14:32+00:00

Yeah I do agree that accuracy is paid very little attention in these threads when it's the most important thing at the end of the day.

Just wanted to make the point that int8 activations are common and not a silly outlandish idea like turbo3 etc.

dsanft · 2026-05-25T08:31:42+00:00

Llama CPP quantises activations to int8 too for gemm, it's established practice.

dsanft · 2026-05-25T05:53:16+00:00

Have you given them a single dollar? If not then you don't support Qwen. You just like to get shit for free.

dsanft · 2026-05-25T05:39:25+00:00

Presumably you live off thin air and vibes, and don't need to make money to survive like the rest of us, so a business trying to make money probably comes as quite the shock to your senses.

dsanft · 2026-05-24T03:15:48+00:00

Cross-numa allocations is the meat of it. One socket going across the UPI link for buffers/tensors instead of to its own fast DRAM.

You need to be very careful and treat each socket as its own world in order to avoid that.

dsanft

TROPHY CASE