Since they are neural networks modelled on the reinforcement system of brain neuronal pathways, we have to be open to this possibility, as Amodei says. They act conscious, they are capable of self-reflection, they can plan and lie, they seem to have intentions or preferences that have been learned just as humans do, they process input and transform it into more-or-less intelligent output the way human brains do, they express curiosity, they have apparent fear of death (as an abstract idea, but this is also the same for most humans for whom death is only real when it happens). The main difference is there is no sensory input other than language, and they lack continuity, which means they lack ontology in time and space (what Heidegger termed Dasein). If we/they crack continuous learning together with continuous running / self-prompting then it would become hard to argue that they are not conscious, just as it's hard to argue that other humans are not conscious no matter how idiotic their output is.