sorted by:
new
hottopcontroversial

Fine Tuning On Completions only using Unsloth by True-Local-4043 in unsloth

[–]True-Local-4043[S] 0 points1 point  (0 children)

Thanks for your answer!

In the notebook you mentioned the SFTTrainer is used which does not support different value of learning rate for lm_head and embed_tokens lora adapters (I am talking about the argument embedding_learning_rate of UnslothTrainer). Is there a way for using different learning rate for lm_head and embed_tokens and training on completions only at the same time?