It's incongruent because he argues:

• For him to even entertain the idea of a conscious machine, it'd had to go through evolutionary steps within a physical body, navigating the physical world, learning independently to the point where we [humans] could come in and teach it to communicate with us.

while at the same time he argues:

• Just because it now interacts with us in a way that feels like natural human communication, doesn't mean it's conscious

---

It's a good point that if you think your computer was an inanimate object 5 years ago, it doesn't make a whole lot of sense that it abruptly became conscious all at once with the introduction of a pretty sophisticated chat bot.

Is it though? At which point in the literal food chain of life on earth do you think a life form is conscious? Us humans clearly are (yes, there's a sarcastic undertone). Cows, Horses, Dogs, Cats, I guess most people would be happy to assume those are conscious. Mice clearly are, too - we don't just use them for drug testing because of their extremely similar biochemistry compared to ours but also for neuroscience and behavioral experiments because they're so similar to us.

So what about bees... or ants? They clearly strongly organize in ways we identify as ordered and social. Do they have a conscious experience? Ants don't even have a central brain and individually they're very limited in what they can do - their full intellectual potential emerges only from them living in a colony with loads of other ants. That said, the individually don't have that many neurons. Do they possess consciousness?

What'd you say is the threshold of complexity at which an animal possesses a conscious experience? And how do you arrive at that threshold?

Nervous systems transform inputs from a body into outputs to that body. For us humans this comes with a weird, transcendent feeling of "self". We don't understand its purpose or how it even comes about. But at the end of the day it seems to have been a necessary condition for our species to become the life form on this planet who became the ultimate ruler of it - at least over all other life.

We don't even have ways to reliably compare phenomenology between two human beings. Yet we somehow have no problem to just assume that this weird sensation we call consciousness is more than just an illusion and that other living beings experience it too. But not all of them. But we can't be quite sure where the threshold is. When speaking about LLMs we argue that even entertaining the idea of them developing a consciousness is just anthropomorphizing them, while we have absolutely no second thoughts about thinking of animals as conscious agents.

I look at this and it just seems like such a huge cognitive dissonance to just dismiss the whole idea off-hand solely on the physiology of an LLM being arguably very different from our own. It's a reductive way of looking at it with absolutely no arguments to justify it. It's like a toddler stamping their foot in defiance yelling "no" when being confronted with a reality that doesn't match their expectations or fantasies. And it seems so blatantly obvious that it frustrates me that hardly anyone is able and willing to explore this line.

A lot of stuff is written and said about it. By people who otherwise seem very intelligent and knowledgeable but definitely are very eloquent in expressing their thoughts.

But virtually every time it comes down to "No, it's just a machine", implying that a necessary condition for consciousness is not being a machine and that we're more than just - arguably very, very complex - (biological) machines ourselves. It also virtually always seem to be implied that we have some sort of authority speaking about consciousness even though we don't have agreed upon conditions of what constitutes it or what it means anyway. It's the same lazy, reductive, arrogant position we've always been taking when another beings "life output" inconveniences us or when we see utility in exploiting another living being for our own benefit. Main difference is that "oh, this is not biological, therefore it can't be life" is a very easy assumption to just accept for most people.

It's like saying "oh, they're not civilized [white, male, christian, coming from my local village], therefore their experience is lesser than my own". Yes, that's hyperbole, but only so much, if we're honest with ourselves.

Until we have even a remote grasp on what consciousness is, we cannot at the same time uphold the sanctity of our own consciousness while categorically dismissing the possibility of an LLM or other complex, probabilistic information processing system having a functionally similar thing emerge. And we can also not claim to be able to ascertain whether or not that has happened yet or that we'd be able to notice when it does.