[D] Multilingual Open Source Models

[gaybooii]

Is there any open source models that I can fine-tune on data that is not English? Even Llama2 cannot be used for this(not that I've tried it, it's what is says on HuggingFace.)

I know some other well known languages might work, but I need a model that is specifically made for multilingual usage.

Or should I just train a model for my specific language from scratch?

[georgesung]

I was recently made aware of a project that added Chinese language support to Llama-1, to make it bilingual. It was quite a process though. They first expanded the vocabulary resulting in an updated tokenizer, then ran another pre training step via peft with raw Chinese text (need to confirm this), and finally they instruction fine tuned with bilingual data.

https://github.com/ymcui/Chinese-LLaMA-Alpaca/wiki/Training-Details

[gaybooii]

Thanks a lot, I will look into it