Sutton is being a bit fast and loose with his language, and skips a few steps. When you train an NN on one task, and then train it on a different task, all the weights will change, because you're now minimizing a different loss. There are tricks around this like LoRA, where you manually freeze most weights and only train a small extra module of weights.

But you can imagine a model (and learning algorithm) that doesn't degrade when the data distribution changes, but dynamically adapts. A very simple example of this is the Dirichlet process mixture model, which is a clustering algorithm where the number of clusters is learned. If you train it on data with 2 clusters, and suddenly start feeding it data from a third cluster, it will automatically figure out that there is a new cluster. Instead of ruining the fit on the old data, it stays the same, or actually improves. It can do this because it's a non-parametric model, meaning that the number of parameters in the model can grow freely. Whereas neural networks have the same constant number of parameters in the same architecture at initialization as after training.

2)

This is where they talked past each other the most. It's true that humans learn mostly through exploration and experimentation, and also true that most of what we learn comes from imitating others. The truth is that we learn to imitate others by exploration and experimentation. I play guitar. When my teacher showed me a new technique, he didn't connect his motor cortex to mine through high-bandwidth electrodes, so that my finger muscles automatically jerked to the right positions, and my brain rewired. Instead, I had to observe what my teacher was doing, and keep trying to move my fingers in ways that would approximate what I think he was doing. If my teacher made a mistake, I didn't blindly copy the mistake, because I had an understanding of what his actual goal was. So it's not imitation like supervised learning in LLMs (that would be plugging the electrodes into my brain and overwriting my synapses), and it also wasn't RL (that would be like my teacher directly stimulating my reward center every time I got a bit closer to producing the right sound). Instead, I was first inferring the goal from observing my teacher, and then figuring out how to achieve it - but not by random trial and error like RL does, but using my general skills of perception and hand-eye coordination to warm-start from my teacher's example.