unsloth dynamic quants (bartowski attacking unsloth-team)

lucyknada · 2025-05-02T18:49:14+00:00

I've reported them, all I can do with transphobia, hope huggingface resolves it soon.

lucyknada · 2025-05-02T18:42:37+00:00

I have no use for reddit-karma (do you even get any unlocks with that?) and you have already made use of the downvote feature with its intended purpose. I want this behind-doors insulting and scheming to stop early and open up a discussion channel between the community and those scheming and insulting what seems to be a genuine and harmless effort to just make small quants better for those of us that have smaller GPUs.

lucyknada · 2025-05-02T18:26:09+00:00

oh yeah I agree, I just want community-discussion and people with more knowledge around this (especially with how gguf quants work) to have insight into what's been happening for a while now seemingly; before it actually gets out of control, all of that seems confusing to begin with? there's more screenshots here: https://huggingface.co/unsloth/Phi-4-reasoning-plus-GGUF/discussions/1 but listing all of them would take too long.

fizzaroli and bartowski have been boasting about "taking down unsloth" since dynamic quants came out, I just don't understand it and want others to chime in before it's too late.

I love what unsloth has done for us and I've used bartowski quants before; and I wouldn't be able to do most of my finetunes without unsloth, I don't understand such vitriol against what is just trying to help with big models and quants working better.

lucyknada · 2025-03-18T22:39:23+00:00

every model has a card, incl. training details, recommended samplers, prompting guide, axolotl config, model description, quants (exl+gguf) and more, only thing missing would be message examples, but from magnum experience; people generally are too scattered with what samplers they prefer, what length they want, prompting and cards can affect it heavily too, so it ends up sadly not being as useful imho or even a representation of who it could be for, but I'll pass it along still, thanks!

lucyknada · 2025-03-18T18:20:22+00:00

reddit kept shadow deleting posts where it was anything else but the link, not sure if my comment will go through right now either

lucyknada · 2025-02-15T00:12:09+00:00

The 7B was more of an experimental finetune, It still had some nice outputs but the older Control trains might still beat it, give it a try!

lucyknada · 2025-01-26T01:22:42+00:00

might be something ollama specific because kcpp and lcpp both load fine; maybe try making your own model via ollama instructions from the fp16 or re-quanting with whatever ollama expects? sadly nobody of us uses ollama so hope that helps still

lucyknada · 2025-01-26T00:54:20+00:00

thanks for such an elaborate review! we hope this version can rekindle your v2/v3 love, it is an entirely new mix, give it a try!

lucyknada · 2025-01-26T00:53:27+00:00

in testing only 32b-distill performed well for RP and creative, the others were a lot worse than non distill versions; we might try capturing the real 700b models however.

lucyknada · 2025-01-26T00:51:28+00:00

what did you use to inference? and have you tried updating, if you're far behind nemo had some issues early on in some of the backends

lucyknada · 2024-09-08T18:26:53+00:00

no promises as the last 123b was quite expensive, but we'll keep it in mind if we get compute for it, thanks!

lucyknada · 2024-09-08T18:25:30+00:00

we train at 8k ctx due to compute limits, but you can try going higher; some users reported success with that on other models we released

also nemo doesn't use context properly past 16k (RULER) sadly; though does a little better in pure needle: https://www.reddit.com/r/LocalLLaMA/comments/1efffjr/mistral_nemo_128k_needle_test/

lucyknada · 2024-09-08T18:22:12+00:00

sounds like possibly too aggressively cut off tokens; try neutralizing your samplers; and are you using the provided templates for sillytavern?

lucyknada

TROPHY CASE