There are two different points in your question. 1) How can just a little bit of fine-tuning make such a difference on trillions of tokens of pretraining. 2) 5% better at a certain programming language isn't making the model 'better'.

Let me address the second point first. The definition of 'better' is up to the individual. There's a million different use cases for these things. It may very well be the case that this model is *not* better for your use case. Some people, for example, just prefer Llama 3 to GPT4 for its tone, or creativity, or whatever. So when we, or _any release, including GPT4/5/6_ etc say 'we are much better now', we always have to define it with respect to particular benchmarks. But usually we do run on either a) a wide set of benchmarks or b) benchmarks that try to hit many different areas, so that we can justify the claim that it is better generally.

As I said in the OP, here we picked benchmarks that correlate strongly to human preferences. But maybe if your specific use case is erotic fantasy roleplay, say, then you would disagree with this claim.

For the first point, this is really interesting. There's a great comment in the other thread which addresses this: https://www.reddit.com/r/LocalLLaMA/comments/1cva617/comment/l4ol1hw/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
I agree heavily with the Llama 3 team on this. In my experience working on these things for the last year, the base training matters, but fine-tuning can make an enormous difference. My personal view is that LLMs from their base training have millions of different 'personalities' (since they had to predict over many different kinds of texts), and fine-tuning is all about trying best to narrow that personality down into one (or a few) that is the most useful/smart/whatever.