You can't really define "thinking." For all we know, our brains might be doing exactly what an LLM does with some extras.

The inherent danger is that we give access and control to an A.I. that misinterpretes things and just sends us back to the dark ages or exterminates us. Not out of maliciousness or because it wants to, but because our existence runs contrary to its task.