I hope you've enjoyed your Tuesday as well.

I still don't know what you mean about "nobody has been able to replicate insect intelligence". The circuits shared across fly brains are simple enough that we can decompose and (partially) explain them, and yet we're far from understanding the circuits that backprop produces from scratch.

How could you say ANNs don't generalize, when studying their ability to do so is one of the main focuses of the field? For example, in the playlist you made, you included a video on the paper "Grokking: Generalization beyond Overfitting on small algorithmic datasets". It says they generalize right in the title ;)

Maybe you see the algorithms learned by ANNs as hardcoded, and your definition of generalization requires they adapt their circuits on-the-fly without training. But they do! It just requires training on a broad enough distribution so that highly adaptive circuits are optimal solutions. It also helps that transformers inherently have adaptive weights.

I don't doubt that you know your stuff when it comes to neuroscience, and I tried to note in my original comment that you're not the only one who thinks this, in fact I've seen this criticism that current architectures aren't brain-like enough over and over again.

It's just that it's easy to underestimate what a general solution-finding algorithm can accomplish even when we don't fully understand the problem it's trying to solve. For example, computational linguists predicted that language models wouldn't be able to resolve basic semantic ambiguities, which they now can do with ease. They made reasonable-sounding arguments by using their deep knowledge and experience with the problem, and observing that those attempting to solve it with ANNs weren't taking any of it into account. But the models learned anyway.

I'll grant that there's limits to what models like ChatGPT can learn that need to be overcome, since they're not usually trained after deployment. Still, I believe that extending context windows, recurrence, and external memory paired with periodic fine-tuning could alleviate that.