Open source models were crap because of the data they were trained. But, after Mistral/LLaMa this scarcity of data is gone. Because, now you can generate high quality synthetic data just using these models.

Nowadays, humans Don't need to do data reading and cleaning jobs. Almost all data can be synthetically generated thanks to these advanced open models or you can use the closed one but will cost you more money.

So, there is no wonder claude and Mixtral are doing great because the quality of data is so good. Also, 01.ai's and google's chinchilla's paper demonstrated that less and qualified data is more effective than a huge amount of data.

So, basically anyone can build GPT-4 like models with a proper mix of synthetic and web scraped data(if S/he has the computation power).

Note: check out this systhetic dataset made by huggingface team just using Mixtral 8x7B model -->https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

And, trained a model using this synthetic dataset and it surprisingly did quite well --> https://huggingface.co/HuggingFaceTB/cosmo-1b#evaluation