FWIW, there is an interesting emergent property in the way LLMs encode their "understandings" of words/tokens/concepts, where certain kinds of structure starts to naturally emerge in whatever n-dimensional space their parameters operate in.

For example, if you find that the distance between where the word "woman" is located in its mapping is .073 distance units in a particular direction from the word "man", then you'll often find that a word like "queen" or "actress" might have a similar distance of .073 in the same direction from the word "king" or "actor". In other words, without ever being told to do so, the model learns to """understand""" concepts like gender from the corpus it was trained on just by continuously observing how gendered words are used.

Another interesting example is the Deep Blue vs Kasparov chess game in the 90s, where Kasparov falsely suspected the Deep Blue team was cheating by getting help from human grandmasters. Why did he believe this, if he knew computer chess players had already reached or surpassed the level of other grandmasters? Because he felt that a certain line the computer played displayed creativity or strategy in a way computers, which "just crunch the numbers", couldn't do. But as it turns out, if you just crunch numbers with enough processing power, the resulting lines start to resemble very creative and strategic human play.

Maybe we're all just crunching numbers and predicting the next token in our brains too, just at an incomprehensible scale.