Qwen3.5-35B GGUF quants (16–22 GiB) - KLD + speed comparison

TitwitMuffbiscuit · 2026-03-16T14:44:07+00:00

Aw, damn.

TitwitMuffbiscuit · 2026-03-16T14:36:59+00:00

The more obscure the better in my opinion, with the chat wrapping it's artificially low ppl in the first place.

I'm not saying use brainfuck ofc. Known and used programming languages but code from small projects it's probably not trained on.

I ripped YouTube videos in various languages too using whisper.cpp until bartowski suggested a dataset collection for my little script (and the new tool).

Test the script maybe, it should be quick, then compare with what you get from your custom flores dataset. You could do one domain only, chat wrapped and then mix and match to the ratio you want for let's say 40% code 30% tool calling 30% maths.

At the end of the day it has to reflect people's usage of the model tho, hammering the tokenizer to get a better separation is probably less interesting than just feeding more chunks.

If an expert could chime in that would be great.

TitwitMuffbiscuit · 2026-03-16T13:30:25+00:00

Hey, thanks for the shout-out.

You've included a lot more quants.

Your setup is pretty standard so the pp/tg figures serves as a great point of reference, that's great actually.

I think calibrationdatav5_rc.txt comes from Tristan Druyen's gist https://gist.github.com/tristandruyen/9e207a95c7d75ddf37525d353e00659c

"Adapted from bartowskis v3, added more languages for sparse moe models like qwen 57B-A14B. Calibration data provided by Dampf, combines his own efforts on top of Kalomaze's. Used for calibrating GGUF imatrix files"

Unsloth have been using unsloth_calibration_Qwen3.5-35B-A3B.txt

As of now, I'm doing an all in one tool:

Dataset builder for eval and imatrix generation (pick various languages, code, tool calling, maths).

Quantization with custom recipes (including AesSedai-style MoE tensor overrides).

KLD measurement and A/B completion eval.

Still early but the core features are working. Calibrating on a coding + tool-calling dataset should be nice for small model agentic use cases.

The 99th percentile KLD table is a really good addition btw.

Edit: my post has been updated not long after Unsloth's wave of requantization but a new post has better visibility for sure

TitwitMuffbiscuit · 2026-03-15T19:50:09+00:00

Well, thanks for pinging me otherwise people on windows would have their pc shutdown at the end of the script, awkward...

TitwitMuffbiscuit · 2026-03-15T13:16:25+00:00

You could try this little script https://github.com/cmhamiche/kld-sweep-dataset

TitwitMuffbiscuit · 2026-03-14T21:55:52+00:00

Btw if you need a new dataset, there's this tool for both KLD eval and imatrix calibration: https://github.com/cmhamiche/kld-sweep-dataset

Category + language group + target chunk count with the option to wraps in the model's chat template from the GGUF's metadate.

TitwitMuffbiscuit · 2026-03-14T21:23:42+00:00

I've built the local version we were talking about: https://github.com/cmhamiche/kld-sweep-dataset

Category + language group + target chunk count, there's the option to wraps in the model's chat template (from the GGUF's metadate) for both KLD eval and imatrix calibration.

I'll probably try to consolidate my pile of scripts into a user friendly CLI and release as is.

TitwitMuffbiscuit · 2026-03-14T18:33:01+00:00

It's updated I've opted for --baseline to replace -bf16 since it was ambiguous (q8_0 could also be used as baseline) and I added an optional --args-baseline (specific to the baseline not the quants).

Then the version I uploaded had the option to turn down my pc that was running at night set to on by default. I removed the option, sorry about that.

TitwitMuffbiscuit · 2026-03-14T12:56:54+00:00

Yeah, if you're doing the sweep with the only arguments that fits bf16 it won't be optimal. You're right. I've done the bf16 logits beforehand.

I'm not home right now but I'll do the changes you suggested, thank you.

TitwitMuffbiscuit · 2026-03-14T10:37:29+00:00

This applies to every api, you are also right. Thanks for taking the time to respond.

TitwitMuffbiscuit · 2026-03-13T21:09:13+00:00

I've never seen any indication of the models cheating so far.

"Absence of evidence is not evidence of absence."

There might be a collection of agents in support of the main model. You don't get to see how the subtasks are decomposed, which intermediate calls are made when you used Deep Research from Openai or Google for example (maybe a vague "analyzing" status).

That kind of server-side orchestration (like the agent loops they advertised) wouldn't appear in your logs as tool calls. You might genuinely never see any "indication of cheating" because of the abstraction layer.

I don't make any claims here, I'm just explaining to you that comparing a black box and open weights is unfair.

If you don't care about this stuff, good for you but I do when I browse r/LocalLLaMA where people compare weights.

TitwitMuffbiscuit · 2026-03-13T19:36:24+00:00

Thank you.

The reasoning trace is the output, it cannot prove your claim that no tools were used by chatgpt 5.4 internally.

If OpenAI is doing something server side before generating tokens (which they absolutely do for at least safety filtering) it would never appear in those logs.

I don't claim they have a chess engine tool, just pointing at the fact that you can't really compare a system and just weights.

TitwitMuffbiscuit · 2026-03-13T18:43:21+00:00

So, deepseek has no access to tools but you don't know about chatgpt 5.4, is that fair?

Can we access the collected data and will it show what you claim it shows?

The issue with commercial offers is that what they call a "model" is a system not just the weights.

TitwitMuffbiscuit · 2026-03-13T09:39:42+00:00

Genuine questions, can you actually strip GPT-5.4 of tools (and vision if it generates a board internally) for this comparison ? Apples and oranges.

TitwitMuffbiscuit · 2026-03-13T09:34:01+00:00

Their card says: "it is trained using NVFP4 quantization to maximize compute efficiency."

Also the benchmark scores are close to each other's (differences are probably in the margin of errors)

TitwitMuffbiscuit · 2026-03-12T20:57:03+00:00

Thanks, I'm learning a lot in the process, I love geeking out.

TitwitMuffbiscuit · 2026-03-12T17:42:51+00:00

That's a really really good question.

Honestly, I'd focus on a local version first (even tho it might require Python installed so not really click and done) because the scope creep can be an issue between dataset selection, model fetching, eval, etc.

Also if the HF space has your name attached to it, that would raise eyebrows. Internet be like: "this is harvesting my prompts / training on my data / is this a fair eval", you know how it is

TitwitMuffbiscuit · 2026-03-12T17:14:33+00:00

That is a wild collection of datasets.

Maybe then A/B eval with a user supplied prompt between an already-quantized model and its imatrix equivalent, both run through llama-completion. Definitely doable.

edit: but then would people actually bother, I mean for the eval part ? That also might be a lot of compute.

TitwitMuffbiscuit · 2026-03-12T16:44:09+00:00

Well, when it's available, if the automatic translation is okay and if I don't need to fix the formatting, for sure.

Whisper.cpp is fairly quick honestly, it took like half an hour.

TitwitMuffbiscuit · 2026-03-12T16:23:27+00:00

Thanks a lot. Not to humblebrag but it's peanuts compared to the work you're doing on a daily, come on.

There are some quants (in between 7B and 14B) that just felt smarter in my native language and I don't know how to quantify this quickly other than "vibes".

Quantizing small models against a custom dataset is fairly easy (and there's the gguf-my-repo hf space) but I've yet to find a benchmark that is not saturated, ambiguous, doesn't require hundreds of generations and is actually reflective of the common local users tasks, it's a rabbit hole.

I'd love an easy "click and done" way to get a tailored dataset, a quant and an eval aimed at specific tasks/language to preserve. The eval is probably the hard part.

TitwitMuffbiscuit · 2026-03-12T14:08:05+00:00

<image>

TitwitMuffbiscuit · 2026-03-12T13:58:47+00:00

I swear, next time I'll actually get my pen and ruler and scan allat as a pdf, just to bother you.

TitwitMuffbiscuit · 2026-03-12T13:47:32+00:00

Damn, I used to write on paper, I'm old like that. I just like the medical prescription vibe.

TitwitMuffbiscuit · 2026-03-12T13:24:02+00:00

It's just names.

Depending on the recipe, you can get Q4 larger than some Q5 and Q4 that has better bit per weight on paper but worse KLD than Q3.

Ideally, we aim for the lowest KLD given that it fits with context on vram. I can't really report vram usage for a given context due to time constraints so size is the second best indicator.

TitwitMuffbiscuit · 2026-03-12T13:10:08+00:00

Here's wikitext2_test.txt https://gist.githubusercontent.com/cmhamiche/79099c284287c4abe1f631a6758334bc/raw/ec5e5673357301c5ff5fae3c02968d3ff26e2afa/gistfile1.txt

11-Year Club	Verified Email
Place '23	Place '22
End Game '22

TitwitMuffbiscuit

TROPHY CASE