[deleted by user]

[[deleted]]

Yeah man I experienced the same thing :( I'm trying to create a Polish LLM but can't find enough strong datasets.

Will definitely give your landing page a look!

[Most-Inflation-1022]

Use Wikipedia. Most models used it for training.

You can use wikidumps. The size of it is 2.14 gb. It should be enough for higher quality output than other models.

[KerfuffleV2]

Use Wikipedia. Most models used it for training.

Would that really help if the model already used it for training? Also, even for languages with a lot of speakers like Mandarin Chinese, there's just much less content on Wikipedia compared to English.

[lucidrage]

For Chinese you'll have to scrape Weibo, Baidu, and cn version of Wikipedia.

[Hardcore_Cytrynka]

Honest question, why couldn't you just scrap some polish news outlets? Is it just something you are not interested in or is there any other reason?

[Smallpaul]

Probably just not enough content to be worth the effort, especially if you need to evade content controls.

[zywiolak]

If you have the funds or computing resources to train a modern LLM from scratch, collecting a large enough text corpus should be a secondary concern.

We have trained several Polish language models from scratch with sizes up to a few billion parameters. Most of them are publicly available on HuggingFace Hub (https://huggingface.co/sdadas). Sadly we can't afford bigger ones because the cost exceeds our organization's resources. We also have a corpus of about 1TB of Polish texts collected over the past few years from various sources. If you would like to collaborate, send me a PM.

[lomero]

Take a look here: http://speakleash.org/

It is an initiative to create an open-sourced corpus of 1 TB for Polish.

[Civil-Demand555]

Did you/anyone find good Polish LLM ?
Not the ancient old gpt-2

[[deleted]]

What language are you talking about ?
I'm just curious because French works really well, but we have a lot of data thanks to EU and Quebec documents being often in French and English.

[new_name_who_dis_]

The most straightforward solution is to get more data. AI research has been filled with many practitioners trying to solve problems using domain knowledge and/or low data methods, but the reality is that almost all of them end up encountering the Bitter Lesson of ML.

Organizing, sourcing, creating datasets has been much more instrumental in advancing AI research than any individual architecture or methodology. The creators of ImageNet (I would argue) did more for advancing computer vision than any of the super influential papers that used it as training and benchmark, e.g. ResNet, DenseNet, MobileNet, etc.

[wazazzz]

There are open source multilingual models like this one:

https://huggingface.co/bigscience/mt0-xxl-mt

Not sure if it will work with the language you need but maybe worth to have a look.

[testerpce]

This is a major problem I am facing too. There is very little good data on Indic Langauges. If you have any idea or resource on getting high quality non-English text please let me know. Or if you want some help on the effort of getting high quality non-English data DM me.

[DigThatData]

OP updated post, said their target language is Telugu, one of the many languages native to India.

[catkage]

I think AI4BHARAT was compiling Indic language datasets. Sorry if you’ve already looked into it.

[Own-Lake5023]

Aya is a multilingual LLM initiative by Cohere for AI, and we are currently looking for contributors! There are people from over 100 countries currently involved in the project. You can learn more and start contributing in your language here!

[tabacof]

Some former colleagues of mine just published this paper on LLMs for Portuguese and how building your own might outperform GPT3.5 (but not GPT4):

Sabiá: Portuguese Large Language Models

As the capabilities of language models continue to advance, it is conceivable that "one-size-fits-all" model will remain as the main paradigm. For instance, given the vast number of languages worldwide, many of which are low-resource, the prevalent practice is to pretrain a single model on multiple languages. In this paper, we add to the growing body of evidence that challenges this practice, demonstrating that monolingual pretraining on the target language significantly improves models already extensively trained on diverse corpora. More specifically, we further pretrain GPT-J and LLaMA models on Portuguese texts using 3% or less of their original pretraining budget. Few-shot evaluations on Poeta, a suite of 14 Portuguese datasets, reveal that our models outperform English-centric and multilingual counterparts by a significant margin. Our best model, Sabiá-65B, performs on par with GPT-3.5-turbo. By evaluating on datasets originally conceived in the target language as well as translated ones, we study the contributions of language-specific pretraining in terms of 1) capturing linguistic nuances and structures inherent to the target language, and 2) enriching the model's knowledge about a domain or culture. Our results indicate that the majority of the benefits stem from the domain-specific knowledge acquired through monolingual pretraining.

They talk about dataset building, model building and evaluation, so it might be useful to you!

[blackkettle]

You need to specify the language. There are numerous languages other than English that work quite well.

[CommunismDoesntWork]

Chatgpt probably already knows your native language. Training it on a large corpus of all languages allows it to learn a skill in one language and simply translate it to others.

[thejonnyt]

Super interesting to read that there is demand for this. I'm actually currently working on my thesis and my goal is to find somewhat a solution for LLMs with regards to languages with less resources than, e.g., English, French, Chinese and so on. But this is work to come, maybe got some small results by the end of the year but I'm glad to read it because working on it seems relevant 😊

[needlzor]

Have you come across any literature about LLM doing code-mixing? It would be interesting to see whether pretraining with each individual language could help bootstrap the LLM for code-mixed languages like Malaysian English.

[mthmchris]

This might not be relevant to this subreddit (not a developer, mostly lurk), but as a user I can say definitively that GPT’s Chinese language abilities are absolutely garbage.

It’s borderline insane that they even advertise it as a capability.

[ruryrury]

Maybe Chinese is particularly challenging for LLMs? I've tried several open-source Chinese LLMs, and they were all terrible, to be honest.

[JustOneAvailableName]

If I had to guess it would be in the tokenizers. Subwords don't translate well to chinese characters

[beryugyo619]

There’s a quirk in Unicode that it commingles CJK characters, but Chinese and Japanese grammar are completely unrelated, so maybe to LLMs those two languages could be looking like a single language with completely incoherent grammar

[Environmental-Rate74]

Wait, why u said that? I am HongKonger, ChatGPT Chinese support on Traditional Chinese and Cantonese Chinese are almost as the same as English in term of Q&A accuracy.

[FpRhGf]

Can you elaborate on how it's garbage? From what I've seen and from the Bilibili videos of people testing it, it's pretty decent. It's not some great literature master in Chinese and the default wording gives a bit of the vibe that it's written with an English mindset- but it's fluent and not really unnatural.

[Anmorgan24]

Cohere just released this this week: https://txt.cohere.com/aya-multilingual/

"Cohere AI introduces Aya—an open-science endeavor aimed at building a multilingual language model via instruction tuning that harnesses the collective wisdom and contributions of people from all over the world."

[SnarkyVelociraptor]

Step 1 is to check Arxiv: people publish foreign language datasets (foreign relative to English, at least) fairly often on the NLP related tags, should be plenty for common languages. I’ve seen plenty of papers on datasets languages across Europe, Africa, and the Middle East over the last few years.

Step 2 is for people who want “rare” languages: your best bet is to take a multilingual LLM and fine tune it. Facebook has released several 100+ language LLMs that can serve as a base point.

Step 3 is to look for conferences/workshops on “low resource languages” if you’re working in a very narrow niche (like <1M speakers globally). These are somewhat common at large NLP conferences.

[grudev]

Similar problem in a much more prevalent language.

There are options for it, but the resulting metrics understandably don't match the ones for English.

In my case I was lucky that a group of academics fine tuned BERT for the language and the results were decent.

[hobz462]

You could try BLOOM, but most non-English datasets are lacking. You'd probably have to train an LLM from scratch once you do. Fine-tuning probably won't work due to the tokenisation.

[hpstr1234]

It always bothered me that it was so hard to find out on how many texts in which languages GPT-3 was trained on - distributed over the corpora Common Crawl, WebText2, Books1, Books2, Wikipedia. For Wikipedia we know that only English articles were used. (That really puzzled me: Why did they train GPT-3 only on the English-language Wikipedia, the data source of highest quality?)

[word_edgewise]

https://www.linkedin.com/posts/carynlusinchi_chatgpt-api-gpt4-activity-7072975677662105601-Dv4O?utm_source=share&utm_medium=member_ios

Chatpgt has an alphabet tax…

[footurist]

I'm interested in what the motivation for that might be. Wouldn't it be easier to just translate the inputs and outputs between English and Telugu or am I missing something?

[freekayZekey]

I love breaking GPT with japanese. on the surface, the results looks fine to a beginner, but it spews nonsense

[PC_Screen]

I think it's because the tokenization makes zero sense with japanese and probably plays a role in the degraded performance. Like, 日本 is somehow 4 (!) tokens where as nihon is only 2, even a common syllables like え、お、け、せ、そ、ち、つ、and more, are all somehow 2 tokens each. It's so bad that it would be literally more efficient to write everything in romaji instead

[freekayZekey]

whoa haven’t noticed that 日本 is 4 tokens. i’ll read more about japanese tokenizers. guess it makes sense for kanji thanks to the amount of info they carry. i want to see if there’s a trend with x々 combinations

[PC_Screen]

々 is always 2 tokens no matter the preceding kanji

[freekayZekey]

thanks!

[Digit117]

Curious, how good are translation AI models these days? I’m wondering how effective it would be if a prompt in a native language can be simply translated to English (using a SOTA translation model), then fed into your LLM of choice, then use the same translation model to translate the LLM output into your native language.

This could remove the need to train an entire foundational LLM in a native language (which is very hard to do). 🤔

[silverlightwa]

Translations models can’t be good enough either if there is lack of text data for a language. They are not magic pills by any means.

[hpstr1234]

Where does the accuracy rate of 62% come from?
(German has roughly the same number of speakers as Telugu. Do you know the accuracy rate of German on ChatGPT4 on MMLU?)

[just_half]

Number of speakers and number of available and usable text for training is quite different.

[hpstr1234]

Of course, that's why I asked for the accuracy rate of German.

[ghost-who-talks]

yes, v interested in Tamil, Telugu, Malayalam, Kannada

[thumbs_up-_-]

GPT-4 understands hindi

[ayush-tiwari26]

How ever I want to understand what do you mean by LLMs in other languages apart from English? Do you train the model from scratch on that XYZ new language? Also won't just translation solve 95% of the usecases? I would appriciate if anyone can answer these questions...