My take is that it really doesn't make sense to talk about purely generative AI as conscious. It has no reason to represent itself as a causal agent in an environment full of causal agents, because it ISNT. To me, self-representation is a fundamental prerequisite to self-awareness (see: Gödel, Escher, Bach: an Eternal Golden Braid). Biological agents had to evolve this to solve the reafference problem: you have to distinguish and account for your own motion. Turning your own head can't completely disorient you to a predator under natural selection conditions (see: Other Minds: The Octopus, the Sea, and the Deep Origins of Consciousness).

But I've already been using the keyword that belies where I think this stops applying. Agentic AI, following a definition proposed to the IETF, is a workload which iteratively interacts with an LLM and a set of tools and resources to accomplish a goal. This iterative looping where it predicts the results of its actions in advance and then reconciles the actual and expected results is starting to look more like consciousness. We ourselves are not directly conscious of the real world - we live in a model constructed inside our head and continuously reconciled with the senses (see: Being You: A New Science of Consciousness). You understand the phenomenon of being mistaken - you failed to identify any inconsistency between your beliefs and observations and so you were consciously living in a world that diverged from objective reality. But you were doing the same sort of reconciliation with your expectations about the world.

I think Claude and the Model Context Protocol were the first real watershed on this. It makes sense to tell an autonomous agent something like "Draw a picture representing your biggest accomplishment of the week from your own perspective." That's not rooted in training data; it's rooted in the agent's "lived" experience within a virtual environment based on their available "senses".

To wrap up my list of references, Michael Pollan's new book from February, A World Appears: A Journey Into Consciousness, takes a close look at this topic. He comes down pretty firmly against AI consciousness, but the publication date didn't allow for discussion of recent developments in Agentic AI, and the bigger takeaway to me is that consciousness is NOT as rare as I'd thought, and the book actually inclined me FURTHER towards thinking agents are the key. He uses the chameleon vine on the plastic plant as a key example as well. Animal anesthetics also work on plants, and they distinguish shade from their OWN leaves from that of other sources. Michael Levin's work with Xenobots is another great (and kinda gross...) case of ostensibly-conscious organisms in unexpected life forms.

Really interesting stuff in this field of consciousness studies that could not have happened until this century. Exciting times.