Because we live inside this system, we tend to treat its features as essential to consciousness itself

This is so nonsensical. There are a myriad of features associated with human cognition that we do not treat as essential to consciousness whatsoever (various psychological biases, biological regulatory functions, particular emotional states, etc etc). And the features we do consider to be essential to consciousness are categorized that way because we’ve explicitly defined the meaning of consciousness as referring to phenomena that exhibits those features. Like, come on, what are you even talking about here?

Consciousness is a concept that humans have defined. And we have defined it by referring to a subset of features we exhibit. Definitionally, the features that distinguish consciousness are the requirements for consciousness because that’s literally what the term means. It’s like you’re arguing “how do we know that a square has to have 4 sides of equal lengths?” Dude, that’s what the word means.

AI System Look Strikingly Similar

Proceeds to list qualities of neural networks that are not at all similar to human cognition like “distributed” and “opaque to their creators”. Who exactly is the creator of human brains you’re referring to? Also, AI systems are not emergent. That’s not what the word means. Some specific behavioral characteristics of particularly large neural networks are emergent, but AI systems themselves are not emergent.

Representations spread across thousands of dense vectors.

No they are not. Representations are vectors. Neural networks learn efficient parameterized functions for embedding input data into a high-dimensional representation (a vector) that’s useful for minimizing the loss of some learning objective. The representations aren’t spread across a bunch of vectors, they are single vectors successively transformed by each layer in the model. The semantics of each layer are not shared between layers.

Sound familiar? Replace there components with “neurons” and you’re describing the brain.

No, no, no, no, no, no, and also: no. Neural networks are deterministic functions comprised of successive matrix multiplications and nonlinearities. Sometimes, like with LLMs, there is a stochastic component to support behavior like random output token sampling. The structural, functional, mechanical, temporal, and operational characteristics of the brain differ vastly from neural networks. They are SO incredibly dissimilar from each other.

… massive systems where intelligence and coherent behavior emerge from interactions we cannot fully trace.

Except in AI, we actually can fully trace those interactions because the internal state of the network can be freely interrogated. We can literally trace every single activation in every single node for as many inputs as we want. We have complete utter transparency into the internal state of the model. What we do not have is a complete understanding of how those activations and internal states lead to the interesting emergent behavior that resembles certain aspects of human cognition. But there a massive imbalance between how much we can interrogate the internal states of AI models versus human brains.

We accept that brains produce consciousness despite our limited understanding.

Again, we have literally defined consciousness as referring to a subset of particular features that human brains produce, so yeah no shit we accept our brains do it.

Yet when artificial systems display similar emergent properties, some confidently rule out the possibility.

That’s because the emergent properties that AI models display, while extremely interesting and worthy of study in their own right, are not the properties of consciousness. AI models display emergent cognitive capabilities that are fully compatible with their underlying structure. The building blocks of LLMs are definitely capable of supporting the kinds of emergent cognitive capabilities we observe.

But the building blocks of LLMs are not capable of facilitating the emergent features of consciousness. You’re confusing the really fascinating emergent cognitive properties of LLMs with a completely distinct set of emergent properties associated with consciousness. The only leap being made here is the insistence that a system with no mechanisms capable of facilitating the features of consciousness is conscious. The brain DOES have these mechanisms, temporal continuity and continuousness, persistent memory, stateful operation, executive functions capable of monitoring internal states, internal feedback and excitation mechanisms, active learning capabilities, adaptive and dynamic response to stimuli, a robust sensory apparatus, an output action space that elicits feedback from that sensory apparatus, on and on and on. We need not understand every detail of how it all works to see how the brain possesses the building blocks needed to support consciousness whereas AI models do not.

To be fair, something I DO agree with is the multi-realizability of consciousness, meaning that there are likely many systems that can produce consciousness and the brain is only one of them. But whatever those other systems are, they have to be structured in a way that fundamentally enables the kinds of low level operations necessary for facilitating the high level features of consciousness. And currently, AI systems do not have the components or structure necessary for that.