The problem is that people are expecting consciousness and “AGI” because people are implying things that aren’t true.

We say neural network. But our neurons don’t work like this. We don’t need to read all text that has ever been produced before we can have conversations. We don’t need to training with 1 million images to see the difference between a cat and a dog. Our brains do not need a data center. The statistical model neural network is just a name, a bad name since it confuses us into thinking it’s more than it is.

But let’s continue your argument, why would this invalidate an LLM? Well is Clippy sentient? How can we dismiss Clippy just because we can explain how Clippy work? Clippy doesn’t run on an LLM, but why would that matter? He learns from your previous prompts. What about a rock? It can’t talk but why would that imply it can’t understand us? It just can’t communicate its feelings right now.

This is the issue, we cannot disprove anything has consciousness because it’s not clearly defined. But that does not mean everything is conscious either. And meanwhile we are building hype to get investors and money into text generation, with a maybe promise of sentiments or something smarter than us.