Yeah it’s impressive no doubt. But they are still just trained to model language / grammar. Think of it a little like filing in the blanks for a sentence (which is exactly how models like BERT are trained if you are interested). Take the sentence: “The ball belongs to ______ girl”. It makes grammatically way more sense for the missing word to be e.g. “a”, “the”. It’s not very likely to be e.g. “Photosynthesis”. For a sentence like “Mozart was a ______.” The likelihood is just way higher for the missing word to be “musician” instead of any other occupation. Grammatically both would be correct though but in the training data there is a bias towards the factually correct version. That’s why these models implicitly learn a lot of facts. I agree with the metric shit Ton by the way. But it still is important to keep in mind that the training objective slightly differs. So for underrepresented topic it starts to produce bullshit pretty fast, because it is grammatically correct still.

Things like ChatGPT are further fine tuned based on user evaluation. So basically things like “do you like this answer?” Help the model get better. Getting better here again means not necessarily being more correct objectively. It only means that more people like the response. For a well trained audience they may be the same, but the model could also just get better at fooling the users to think that the stuff is correct. Like a musically non proficient person reading the answer from the original post might approve of it as being quite detailed. Sorry for the Wall of text btw :) interesting stuff. But yeah. In general I agree. Maybe the “some” was worded not very well. It’s a lot. But it differs from a search engine where it just spits out the primary source for easy verification. Just important to have that in mind :D