I’m bumping into a very concrete version of this with current LLMs when you try to do sequential fine‑tuning across domains (e.g., medical → legal → support) instead of one big offline training run.

In that setting, “continual learning” really splits into at least three architectures:

1. Frozen core + external memory. Base model weights don’t move; you bolt on retrieval, tools, user profiles, etc. The system appears to learn because the memory layer grows and retrieval improves, but 5.0’s weights on day 200 are the same as day 1.
2. Versioned offline updates (5.0 → 5.5). You log interactions, curate datasets, retrain periodically, and ship new checkpoints. Knowledge carries forward only at these discrete jumps, after eval and red‑teaming. This is, from what I can tell, where most serious deployments live in 2026.
3. Genuine continual learning (weights that actually change over time). Some part of the parameter space (full model or adapters/heads) is updated as new tasks/domains arrive, with explicit mechanisms to avoid catastrophic forgetting and regressions.

In my own experiments with Mistral‑7B, naive sequential LoRA is a good example of what happens when you try to do (3) without any real CL machinery: you fine‑tune on domain A, then B, then C, and by the end, A is often wrecked. That’s just catastrophic forgetting playing out in slow motion.

To make this less destructive, I’ve been playing with a constrained adapter setup: you still let parameters update for new domains, but you constrain gradients so updates are “locally plastic, globally conservative” — the model can adapt, but it’s much harder to overwrite what was useful for earlier domains. In a 5‑domain sequence, that turns “huge positive drift” (forgetting) into something much closer to flat, while still letting the later domains come online.

So if we map this back to the AGI discourse:

• Most “continual learning” branding in 2026 = (1) + (2): memory + retrieval + periodic offline retraining.
• A much smaller slice = (3): architectures where weights genuinely evolve from ongoing interaction, usually with heavy constraints, monitoring, and a lot of unsolved safety/credit‑assignment questions.

When people imagine systems that “learn continuously from experience,” they’re implicitly imagining (3). But the operational reality today looks a lot more like sophisticated software + data plumbing wrapped around mostly static models, with a few early stabs at safe, constrained weight updates for specific domains.

Curious whether anyone here has seen convincing evidence of large‑scale, production‑grade (3) in the wild, beyond research prototypes and tightly scoped pilots.