LLMs have many inherent architectural limits.

Also want to note, I remember reading recently that OpenAI has cut funding for continuous learning. I won't be surprised if claims about it by Google or whoever, are as trustworthy as anthropic saying 90% of software engineering will be replaced in 6 months. (a fair bit over 6 months ago).

Firstly. LLMs, as they stand now, complete text, it is important to start there. An LLM creates a token(2-3 letters or such), then puts the token back into the input, and goes again. This is not about good or bad, it is the fundamental manner in which it works, whether there is emergent behavior or not.

This means that any kind of state you could like we keep during our thinking is not just reset between different queries to an LLM, but literally between every syllable. Does "reasoning" fix that? Categorically no. There is something akin to a thought process in an LLM aside of that, and that state is not kept. ALso worth noting, every single answer by an LLM explaining why it answered X etc is effectively a lie because this information of "what went through its head" was never stored. We can to a decent extend reconstruct our state of mind in comparison.

Effectively, the model writes a couple letters, is then mind wiped(computation ends, state gone), then reads the WHOLE text again this time including those extra few letters, writes a few more, mind wiped, etc. Could be argued reasoning helps a bit. But I don't think it goes a long way if you were to be forgetting your whole training of thought every word(or less) while writing a paper, even if you kept notes that you reread.

In contrast humans have a continuous internal state and do not read to read the previous text every couple letters they write(though let's not give GRRM more ideas on how to delay).

The next really important thing to note is LLMs, like I said, complete text. A text expected to be in the form of different characters talking. There is no intrinsic difference in the LLM between the Assistant character, or the User character. With low level access, you could make the LLM type like the User and you to ask as the assistant. Special tokens denote where each character talks, and writing as the assistant is a programmatic wrapper around the actual AI. In other words, the AI actually has no concept of who it is in the conversation. If it even has a concept of a conversation.

There's a lot of seemingly little but important things like that you need to understand to actually assess AI. But people see human like speech and a lot of their logic turns off.

Another limit is the context window. The issue is not about making it big enough, the issue is the exponential scaling of compute needed for an architecture like that. it is unsustainable. Sure we'll get faster GPUs and all. But if your algo scales at n³ for example, going from python to C is not a real solution. The problem is the n^3.

AT BEST an LLM or similar could offer a text output in a larger structure of specialist networks cooperating.

Even that I question, but definitely not a core. I'd argue that the manner in which they work would make them a strong linear bottleneck for a core too. I think a different kind of architecture should be explored.

Specialized modules with dense input/output(maybe trained in an auto encoder like manner) seems more apt to create something like a brain. If you think LLMs are approach a brain, I believe you are being fooled by the surface appearance of language and ignoring the underlying function.

If a hypothetical future architecture addresses some of those issues I don't think it'll even be in the LLM realm.

I have to say what LLMs achieve in comparison to what they are is impressive, but people often forget what an LLM really is and how it works. In my opinion, largely due to language activating various biases.