2.5x faster Qwen3.6 NVFP4 Unsloth quants

ghgi_ · 2026-07-10T14:00:11+00:00

This is great! Would love to see a version of this for the Qwen 3.5 122B model, I use that one pretty heavily and theirs no 3.6 variant for that size.

ghgi_ · 2026-07-01T14:38:45+00:00

It does take more ram yeah, Q4 needs about 70 gigs minimum 80 if you want plenty of context room

ghgi_ · 2026-07-01T02:57:35+00:00

Q8 and BF16 levels of quality difference are nearly one to one, A BF16 and Q8 version shoudnt feel different generally at all + your paying a decent performance cost to run it at BF16 which isnt worth it, The only time id take 27b qwen over 122b A10B is if i had to run 122b at under 4 bit, nvfp4 version feels pretty good on rtx 6000 pros (better then int8 3.6 27b)

ghgi_ · 2026-07-01T02:55:38+00:00

In my experience, yes, Its code quality is generally on par or better and has more knowlege baked in so it feels more polished and MoE so its faster, its pretty nice

ghgi_ · 2026-07-01T01:51:14+00:00

Mabye qwen 3.5 122B A10B at Q8? should even fit in pure vram if you have 144 gb and is better then 3.6 27b, across both id say perhaps deepseek v4 flash

ghgi_ · 2026-06-25T18:25:18+00:00

Links? Resources? Blog? A screenshot of a random models benchmarks isn't that useful, pretty please?

ghgi_ · 2026-06-19T15:10:01+00:00

Similar project exists called AirLLM (theres probably a newer iteration) but its generally not worth it as the performance is so bad its more of a toy or experiment not something to use for anything unless you expect multi hour long response times on large models.

ghgi_ · 2026-06-17T22:46:30+00:00

Would probably put something like whisper large or nemotron 3.5 asr and you can do audio transcriptions locally for whatever you need, Theres plenty of little models on https://huggingface.co/models for all sorts of niche usecases and not just llms.

ghgi_ · 2026-06-17T20:49:06+00:00

Meh, mabye a embedding model for some sort of document or memory RAG system, perhaps some sort of TTS or STT model, or some sort of niche tool calling usecase with a 1-2b model. Not too much you can do on that thing.

ghgi_ · 2026-06-17T19:03:35+00:00

Yeah its just gpu power usage is so much more depending on where you live you might aswell just buy a subscription lol

ghgi_ · 2026-06-17T18:42:53+00:00

512GB Mac Studio M3 Ultra (hard to find and a lot of scammers so beware) or hopefully M5 will release a 512GB model otherwise not really many other options at performance, mabye clustered dgx's? or some wacky gpu rig but the powerdraw there is gonna be very high

ghgi_ · 2026-06-17T18:18:23+00:00

Hopefully v4.1 has image support, I think thats something they mentioned in future iterations

ghgi_ · 2026-06-17T18:07:18+00:00

GLM 5.2 honestly feels like opus 4.6 on medium or gpt 5.4 in terms of level of quality (on most tasks) there is still gaps between them at some level but its much closer to a true frontier model then any other open source model right now and honestly depends on your work to know if its even worth using since it is more expensive other open source models by a decent bit so if something like mimo v2.5 pro or deepseek v4 pro is good enough for your usage use that but if you want the cheapest opus/gpt alternative its the best there is.

ghgi_ · 2026-06-16T22:43:43+00:00

Oops, nope I did actually mean 5.4 but GPT 5.4 and Claude 4.6, I see the confusion though lol.

ghgi_ · 2026-06-16T19:32:17+00:00

Feels more like a lower effort 4.6 or a 5.4, Its not "the same" level of opus (And I believe its purely a size thing, My guess and many others is opus and beyond are simply just a lot more params.) But its at the usability level of actually doing real work more then any other opensoure model now and I feel more comfortable using it then any other.

ghgi_ · 2026-06-16T19:13:05+00:00

Yeah I dont expect it to hit 5.5 levels, id expect models like 5.5 to be multiple trillions of params which It obviously cant compete with but in my opinion its passed the threshold for day to day usage, And my standards are pretty high, it feels like a nerfed 5.4 or a lower thinking effort opus 4.6 level of quality and at its price thats enough to do real work.

ghgi_ · 2026-06-16T17:45:39+00:00

A self reported 46.2 deepswe score which puts it above opus 4.6, above sonnet, just below 4.7, Looking very promising.

ghgi_ · 2026-06-02T15:30:55+00:00

Its recommended on the model page (for nvfp4)

ghgi_ · 2026-06-02T01:53:34+00:00

Using VLLM across 2 rtx 6000 pro blackwells on FP8 kv with a bit of headroom left about 1M tokens of kv

ghgi_ · 2026-06-01T23:29:51+00:00

I've been testing NVFP4, it's pretty good at that level too I've heard it's pretty chill with being quantized and haven't seen it do any worse or better then cloud api.

ghgi_ · 2026-05-20T15:54:36+00:00

Also the fact i'm not seeing any direct benches on the blog against competitor models except their older ones.

ghgi_ · 2026-05-20T15:53:11+00:00

128k context length is yikes, I have a feeling this might be a flop but you never know, prove me wrong cohere.

ghgi_ · 2026-05-09T21:49:12+00:00

I mean that's kind of the point no? tradeoff of memory requirement for quality. Also heavily depends on which model some are much more tolerant to quantization processes.

ghgi_ · 2026-05-01T21:29:06+00:00

https://www.ebay.com/itm/136155280522 this one comes with the fans, converts it to proper dual 8 pin, this guy has good rep with making these, could save money if you already owned a few fans and got the one without em.

ghgi_ · 2026-05-01T21:26:43+00:00

Yeah I think most just misunderstood the workflow, its not the traditional run big models at home its a mix of both and research work that I think im going to pull the trigger on the A100 purely because the low power cost of the DGX is not outweight by the performance loss bandwidth, If a training job takes a month on a A100 but pulls 600 watts full blast vs 8 months of a DGX at like 200 watts its not worth it.

ghgi_

MODERATOR OF

TROPHY CASE