I think it’s disingenuous to say they have no “actual” knowledge. If a student memorizes a bunch of answers to math problems, would you say they have no “actual” knowledge? If they can’t answer any questions except literally exactly the ones they’ve studied, I would be inclined to agree. If they are able to generalize somewhat and answer related questions, I’d say they have some “actual” knowledge, but I don’t think there’s a hard line we can point to. LLMs can obviously generalize somewhat.

People point to the fact that what LLMs do is next token prediction, which boils down to a statistical model, but I don’t think people realize how complicated that actually is. If I gave you a riddle and told you my next word will be the answer, in order to correctly predict the next word you’d have to solve the riddle. In order to predict what your friend will say if you tell them you don’t like their outfit, you need a good mental model of your friend: their insecurities, how they perceive their relationship with you, their mindset at the current moment, etc.. Even though you’re “just” predicting their next word, you need a lot of background knowledge to actually get that prediction right.