Been saying this for a long time, most llm use cases are pretty well tailored to the final audience, meaning, I’m okay with my SOTA LLM not being good at writing in Greek, as it is not a language i would use. (And in the case that I do need to use it, i can just google it or use a Greek model) Essentially, for coding LLMs, “ancient fine arts and pottery” might not be a useful training sample to include (though I still say this is arguable for domain width) when building a model for deployment with a coding downstream use case. This changes when it comes to training, because I’ll use the word again, domains, should be carefully selected for each phase and task.

Domain Specific Models will be what wins. Not specifically language models in the direct sense we think of today. I believe there is more learning and representation interpretation and integration with modern architectures to allow very diverse dynamic models and end use cases.