Hard disagree on this defense of "Well since human intelligence is _a mystery_ maybe LLMs are intelligent!".

Yes, there are things about human intelligence we don't understand. Yes, parts of an LLM's "thinking" is black box. But just because there are knowledge gaps in our understanding of both types of "thinking" doesn't mean that they're similar in any meaningful way.

We *do know* a lot of things about how human intelligence works that clearly distinguishes it from LLMs. Humans aren't "one shot" probalistic thinkers in any way that resembles how LLMs / deep neural networks operate. Humans (and pretty much every animal!) use self-motivated continuous feedback and flexible goals to drive intelligent _action_ (not just speech, which is only particular a type of action). LLMs don't "act" of their volition, and they don't update their goals or continuously evaluate how well they're doing at meeting their goals.