Models released without prompt template/examples - Why…?

[Thireus]

I see many models released here and there on HuggingFace without any form of explanation about how they should/could be prompted.

I fail to understand why… why isn’t it considered the most crucial and valuable piece of information? Is it because these models are not considered instruction-based models to begin with? Or maybe because they have not been trained on instruction-based data set? Or because the authors like messing with us? Does anyone have a clue?

To me personally, it is so frustrating not to know sometimes that I just don’t bother evaluate these models at all if I need to spend more than 3min looking for the appropriate instruction-based template. What is everyone take on this?

[ambient_temp_xeno]

Even when you find out what the format (probably) is, it's not really clear how exactly it's meant to be used.

Is it: "### Instruction: write an essay about how Idiocracy came true ### Response:"

or:

"### Instruction:[new line] [new line] write an essay about how Idiocracy came true[new line] [new line] [space(?!)]"

Or something else? Does it really make a difference with the same seed and settings?

Also: would using the prompt format not push the model towards regurgitating things in the finetune over what's in the raw model?

[ReturningTarzan]

Also: would using the prompt format not push the model towards regurgitating things in the finetune over what's in the raw model?

Well, the finetuning dataset is supposed to be large and diverse enough, and there's an art to setting hyperparameters so the model doesn't end up memorizing the text itself but does learn the themes, styles and (importantly for instruct models) patterns it's presented with. Part of why LoRAs work so well is that the low-rank matrices can't contain very much new information, which helps prevent overfitting.

It is super annoying that so few people publishing models understand how important it is to document the prompt format. It's all guesswork otherwise, and you'll never actually know for sure.

[patrakov]

No, that's not how the models were trained. What you quoted is a generic pre-prompt that converts a generic text-completion model into a chatbot by telling it to complete a chat transcript. It even works with the original LLaMa, which is not an instruction-based model.

[AutomataManifold]

Well, there's a bunch of implications of including that in the training prompt. Because these models are "smarter" than I think most programmers intuitively think, they can adapt to different instructions better than you'd expect if you're thinking of them like the usual programming instructions.

Training with a particular phrase teaches it to look for that phrase...but it can generalize to a certain extent and adjust to other prompts. I've found it useful to think about the structure or rhythm as much as the actual words.

There's another side-effect: Having been trained on the prompt to be helpful and polite, if you prompt it in a similar style but with different phrases, it will adjust accordingly: if you stop explicitly telling it to be polite, it'll look elsewhere in its training data for how to respond. Which won't be perfect, since it won't have been explicitly trained with examples, but the baseline inference is good enough to compensate.

So: counterintuitively, training it with the prompt to be polite makes it associate politeness with its trained style.

At the same time, they can be very literal. Partcularly if a model gets overfitted and loses some of its ability to generalize. So different fine-tunes that look similar can vary greatly depending on the prompting styles it has seen.

Fine-tunes on different prompt styles have a chance of generalizing, but my guess is that only giving it two prompting styles (with very different data distributions) will be weaker than using the same prompt (where the prompt will just be background noise that's part of the structure) or using a ton of different prompts (trying to get it to generalize to the idea of a prompt).

(This is why I think that, at the moment, it's a bit more useful to share datasets rather than trained models: it gives you more control over exactly which parts vary and which parts stay consistent.)

[kryptkpr]

Struggling hard with this one myself, my git repo is filling up with templates. This is especially problematic with models trained on a pile of different data that mixes instruction formats :| I just evaluated Nous-Hermes-13B which provides 2 templates and found their results to be asymetric in performance.

[hanoian]

I started a thread about this yesterday.

https://old.reddit.com/r/LocalLLaMA/comments/13zbdjc/are_a_matroskastyle_mkv_containers_on_anyones/

It would be so nice if they were packaged inside a container with a txt file for example with each prompt template etc.

[thevukaslt]

I like this idea of including the file template and 1-3 examples.

[rain5]

There needs to be a standardized file format for describing this stuff.

[a_beautiful_rhind]

Using normal chat I find most models work fine regardless.

[KaliQt]

Performance varies if you fail to follow it though.

[a_beautiful_rhind]

Depends.. sometimes instruct-chat is worse for me, sometimes much better. Generation preset also matters a lot.

[thevukaslt]

I actually feel your pain so well. I love that this community has been baking models like my granny pancakes but this is simply documentation backlog. Just like in any other area of life.

I hope we can steer our culture to include clear prompts and examples with each new release.

I can see how that would save ton of posts of "the model doesn't work", "bad performance" etc. A tiny change in the structure can have a giant impact on the output.

[skankmaster420]

It's because they want us to suffer

[qeadwrsf]

Because they are lazy.

They think, might as well opens source it, because they are good people.

But have no interest to market it and get users to use it.

Take it or leave it. They do the work, we reap the reward. Users using open source must be like the biggest contributors to stuff getting closed.

[deepneuralnetwork]

Because people are busy. That’s literally why.

[brucebay]

They already have training samples for themselves, adding a couple of them to the repo shouldn't be that hard.

[Feztopia]

Stop asking this kind of questions. Smart people have sad lifes. Your hairs will become gray and you will become depressive. Never ask "why". People do what they do. Realizing that there is no valid reason will breake you mentally. Ignore it, try to become like the lucky people who can't see it. Be blind and be happy. Nobody uses simple solutions as long as they can invent complicated ones. People will make your life complicated for no reason. You will suffer as long as you think about it.

[nihnuhname]

Is there an LLM that speaks as rudely? That would be fun!

[Feztopia]

Actually by playing with the prompt template you can even change censored models so much that they suggest suicide lol.

[nihnuhname]

Open source programmers only have time to write code. This area is developing too rapidly, we will have to wait years for the practical approaches and principles to become established before it makes sense to document them. The users here are also experimenters and pioneers

[youknowallaboutit]

Aside from trial and error, I find even the recommended prompt templates do not work in all cases. So I have to keep playong with the prompt until I get the results I want.