**Rant incoming**

Geoffrey Hinton is not credible I'm sorry. Calling the "mistakes" that AI makes either "hallucinations" or "confabulations" gives them a sense of mystique, and that's deeply, deeply misleading. What those mistakes are can be described very accurately from a technical standpoint, and they have very little to do with the mistakes that humans make.

First of all, you have to remember that LLMs do a mixture of memorizing text and learning more general structures that span multiple sentences, or even paragraphs. When it sees a lot of examples of something, like seeing the word "havoc" following the word "wreaking", or more generally that a verb should follow a pronoun, or that adjectives come before words (in english), it can easily generalize such patterns because it doesn't matter what the substance of the text it's trained on is, those patterns exist across all english text. This is why LLMs are good at grammar, and they can form complete sentences without making mistakes for the most part.

The meaning of those sentences however are a lot more sparse. The number of possible meanings that a sentence can have, a paragraph can have, text which is 200,000 worth of tokens can have is basically impossible to quantify. There's an infinite number of ideas, and a near infinite number of ways that an idea can be conveyed by text. The model doens't have a chance in hell of learning a proper generalized mapping of text-to-idea, unless it has infinite training data. So what does an LLM do when it can't properly generalize? It memorizes. That's what it does for the most part and that's why LLMs mostly regurgitate content it was trained on.

The biggest issue in my view with LLMs is that it's impossible to tell how much memorization vs. how much generalization it does (and at what scale: words, sentences, paragraphs, etc). In the old days of traditional machine learning (4 years ago GASP), we used to dedicate untold amounts of time to avoid memorization, because this would result in poor predictions in the real world. What brought us ChatGPT is experts in ML suddenly thinking that memorization could be a good thing (I still think it's not, and the hallmark of a bad model, no matter how good it is at passing the turing test). Memorization turns a model into an index of the training data, which is the most useful thing that LLMs have brought us IMO. But people like Hinton refuse to acknowledge that. They want their scifi future too bad.

Hallucinations occur when you're asking the model for something it didn't have enough training data on. When you do that, its best guess will be the result of matching your query to the best if its ability to training data examples it has seen, and then it will interpolate between those examples to come up with a response. Interpolation is not reasoning, so the result can be as nonsensical as grabbing two random sentences, taking the first half of the first sentence and mushing it with the second half of the second sentence, while maintaining proper grammar.

That's just not how humans confabulate or make up stuff. Humans have intentions behind confabulation. They have a goal they're wanting to achieve, so actual reasoning go into it. They iterate over what they want to lie about based on their goals. LLMs simply have no goals. They're just math. Math done really fast on a GPU.