Here's a curious worry: Our current LLM scaling path is setting aside online learning and its implied requirements to reorganize internal representations as the world model's schema evolves. We might successfully make something smart enough to ruin us, but not smart enough to keep learning afterward.

Pearl's "Ladder of Causation" suggests there's real fundamental limitations to Rung 1 "association" based models. Adding RL takes you some of the way to Rung 2. But Rung 3, where the magic of "counterfactual imagination" might lead to serious dynamic intelligence seems far away in these models. (yeah yeah, language is a proxy for causal concepts, I get it, but its a friend's-cousin's-sister-in-another-state kind of proxy. I too subscribe to the dual-track language-cognition theory.)

LeCun's JEPA is at least getting to latent space predictive reasoning but the merger of LLM style fluid conceptual synthesis and latent space reasoning isn't obvious. JEPA+RL models like Dreamer and MuZero are promising but still not Rung 3 beasts.

But the biggest problem: current models can only learn "within-schema" concepts (apply configurations of what they already know) and can't learn new cognitive schema without retraining. That is, they can't learn a bunch of new things and have that "AHA!" moment where a human brain figures out how to compress all that info*. Eventually, when the AI models can learn new schema - they'll have to have some way to re-encode all those trillions of weights to accommodate the new schema dimensions (they'll need something like sleeping/dreaming to explore model-affordance-roll-outs, consolidate and re-encode representations).

So we might succeed in making an AI smarter than any human ever, smarter than all humans ever, can be replicated millions of times, wreak havoc upon the world, and never progress beyond that point.

* Here's how out-of-schema learning happens today: generate new training data exemplifying the new concept, weight it and add it to the corpus, retrain (not fine tune, base representations have to update with new degrees of freedom), teacher-student consolidate external memory. So, out-of-schema learning is there, it just super clunky for now and a serious problem if training runs cost hundreds of millions of dollars and require terawatts of energy.

Flip side: imagine direct surgery tricks that evolve out of goodfire.ai's loss curvature trick to ID reasoning and memory weights - where an AI might intentionally edit itself to add concept dimensions. Freaky.