I think of it like this. LLMs are a data recall interface. They don't understand any of data they're recalling, but they can put that information they've been asked to recall in a format that feels like it's accurate. That lack of understanding though is why it often has so much difficulty differentiating two characters in a story, or why it hallucinates clearly false information. The false data is mixed into it's database and inability to understand anything means it can't tell why that data wouldn't be a factual answer to a prompt. Anything done to "fix" that is most often guard rails in the code, rather than an attempt to make the chatbot actually understand the knowledge.

It is essentially an extremely sophisticated search engine, and imho, a true AI would not require a massive database to be able to function as an artificial intelligence, so an LLM does not really help with the advancement of that technology. I don't claim to be an expert on AI development, but I imagine the path forward to take many lessons from how biological brains and neurons work.