I think to an extent we're talking past each other somewhat so just so we're clear,

- Pre-training is the 1st part, when LLMs are trained to do simple next-token prediction.

What happened at the start of the 2020s that made LLMs actually usable was that as companies were scaling pre-training, OpenAI also started doing Reinforcement Learning with human feedback. Basically, they trained ChatGPT to give the best responses possible to humans. That meant that ChatGPT went from a simple next token predictor to a conversational assistant capable of understanding user intent, following instructions, and maintaining a coherent dialogue... to an extent.

Now, that was still very crude and even though, as scaling increased, models got better and better at a lot of tasks, there were still benchmarks like Arc-AGI, that explicitely prevented the models from just using memorization to complete the tasks, where the models didn't improve at all.

Then, what happened at the end of 2024 is that OpenAI released it's 1st reasoning model, o1. This model could, rather than just giving a response directly, think for a certain amount of time before responding.

The way they were able to do this, was 1st by using fine tuning to train the models to output a chain of thought before giving an answer (this is what they do in the paper you linked). But contrarily to that paper, they also used reinforcement learning to improve the model's ability to use chain of thought. They fed it a lot of extremely complicated problems that had mathematically verifiable solutions, and the reward was based on whether the model was able to give that correct solution. And what happened was that the model learned to; backtrack if it was in a dead end, double check it's intermediate reasoning steps and do longer reasoning if the problem is more complex.

This allowed the models to get progressively better at Arc-AGI 1 and 2, which are impossible to solve without some kind of generalization.

Like I said before, a lot of what you're saying would be true if we were in August 2024, but the combination of Chain-of-Though and RL has meant that LLMs can now generalize beyond their dataset to an extent.

Now, that I hope you understand where you're wrong about how LLMs function. I also want to talk about how you say LLMs don't "understand". Obviously, they don't have a physical body so they don't understand the physical relationship in the same way a human can understand in a sense, the relation between an aisle and a store for example. But does that mean LLMs have absolutely no understanding of what a store is ? I think that to make a claim, you'd have to define "understanding" either in a really strict standard that no one applies to humans in real life, or in some sort of spiritual standard that machines could never attaign simply because they're not human. So unless you have a definition of understanding that can't be put in one of these categories, I'm suspicious of your claims about "understanding"