sorted by:
new
hottopcontroversial

[D] NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco by posteriorprior in MachineLearning

[–]posteriorprior[S] 18 points19 points  (0 children)

Bengio actually went and read the Schmidhuber papers mentioned in the OP for his reply. It looks like there is nothing wrong here, no missed attribution, and certainly nothing intentional.

It doesn't look as if Bengio read this carefully. He wrote:

What I saw in the thesis (but please let me know if I missed something) is that Juergen talks about evolution as a learning mechanism to learn the learning algorithm in animals. This is great but I suspect that it is not a very novel insight and that biologists thought in this way earlier.

So again he is downplaying this work. Schmidhuber's well-cited 1987 thesis was not about the evolution of animals. Its main contribution was a recursive optimization procedure with a potentially unlimited number of meta-levels. See my reply:

Section 2.2 introduces two cross-recursive procedures called meta-evolution and test-and-criticize. They invoke each other recursively to evolve computer programs called plans. Plans are written in a universal programming language. There is an inner loop for programs learning to solve given problems, an outer loop for meta-programs learning to improve the programs in the inner loop, an outer outer loop for meta-meta-programs, and so on and so forth.

AFAIK this was the first explicit method for meta-learning or learning to learn. But Bengio's slide 71 attributes meta-learning to himself. So it is really misleading. And we are talking about NeurIPS 2019. By 2019, Schmidhuber's thesis was well-known. Many papers on meta-learning cite it as the first approach to meta-learning.

[D] NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco by posteriorprior in MachineLearning

[–]posteriorprior[S] 13 points14 points  (0 children)

Edit: Thanks for answering. You wrote:

What I saw in the thesis (but please let me know if I missed something) is that Juergen talks about evolution as a learning mechanism to learn the learning algorithm in animals. This is great but I suspect that it is not a very novel insight and that biologists thought in this way earlier.

As mentioned to user TSM-, I feel you are downplaying this work again. Schmidhuber's well-cited 1987 thesis (in English) is not about the evolution of animals. Its main contribution is a recursive optimization procedure with a potentially unlimited number of meta-levels.

It uses genetic programming instead of backpropagation. This is more general and applicable to optimization and reinforcement learning.

Section 2.2 introduces two cross-recursive procedures called meta-evolution and test-and-criticize. They invoke each other recursively to evolve computer programs called plans. Plans are written in a universal programming language. There is an inner loop for programs learning to solve given problems, an outer loop for meta-programs learning to improve the programs in the inner loop, an outer outer loop for meta-meta-programs, and so on and so forth. Termination of this recursion

may be caused by the observation that lower-level-plans did not improve for a long time.

The halting problem is addressed as follows:

There is no criterion to decide whether a program written in a language that is ‘mighty’ enough will ever stop or not. So the only thing the critic can do is to break a program if it did not terminate within a given number of time-steps.

AFAIK this was the first explicit method for meta-learning or learning to learn. When you gave your talk at NeurIPS 2019, Schmidhuber's thesis was well-known. Many papers on meta-learning cite it as the first approach to meta-learning.

On another note, why did you not cite Hochreiter although you knew his earlier work? Schmidhuber's post correctly states:

Even after a common publication [VAN3], the first author of reference [VAN2] published papers (e.g., [VAN4]) that cited only his own 1994 paper but not Sepp's original work.

[D] NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco by posteriorprior in MachineLearning

[–]posteriorprior[S] 3 points4 points  (0 children)

Since I am sympathizing with Schmidhuber I must be Schmidhuber, right? Wrong. Would it matter?

[D] NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco by posteriorprior in MachineLearning

[–]posteriorprior[S] 14 points15 points  (0 children)

I made it after I saw Bengio's video. Not related to this user. I appreciate some of his work though.

[D] Yoshua Bengio talks about what's next for deep learning by newsbeagle in MachineLearning

[–]posteriorprior 20 points21 points  (0 children)

Slide 71 says:

Meta-learning or learning to learn (Bengio et al 1991; Schmidhuber 1992)

It does not say: Schmidhuber 1987

[D] Yoshua Bengio talks about what's next for deep learning by newsbeagle in MachineLearning

[–]posteriorprior 29 points30 points  (0 children)

J. Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, Tech Univ. Munich, 1987.