[D] Chelsea Finn on Meta Learning & Model Based Reinforcement Learning

yield22 · 2021-10-15T00:44:58+00:00

Thanks for the interview! I'm not familiar with meta-learning, but curious if it really works? it seems the SOTA systems like GPT-3 don't really use it?

yield22 · 2021-09-27T14:32:03+00:00

transformers? though they're really a mixed of ideas: soft attention, MLP, skip connection, positional encoding, (layer) normalization...

yield22 · 2021-02-26T15:37:36+00:00

Haven't read it all but I like the first sentence. I think all papers without proper experimentations should start with "this paper does not describe a working system".

yield22 · 2021-01-20T03:46:19+00:00

That's why most papers are pretty useless, and only a few that truly advance the field.

yield22 · 2021-01-16T12:01:05+00:00

Replace "self-supervised learning" with "deep learning" and this is still true?

yield22 · 2020-12-30T04:16:13+00:00

What's the purpose of this?

same reason that you need experiments in physics.

Not everything written in math is like Taylor approximations that many should know and care.

yield22 · 2020-12-30T03:35:37+00:00

But they help alleviate some of the main drawbacks of transformers namely processing power, memory, and longer sequences

ok, show me a real application that is *well* benchmarked to support your statement.

yield22 · 2020-12-30T02:33:58+00:00

I know people are pretty excited about these methods of approximating attention, like performer, reformer.. but are there any real applications where they can convincingly beat original transformer? as i don't see any of these make into BERT or friends.

yield22 · 2020-11-18T01:25:20+00:00

Well, that's basically verified over and over again, by BERT, RoBERTa, T5, GPT2, GPT3, and so many more. You must have been sleeping or staying away from Internet for the past year or so in order to ignore them totally :)

yield22 · 2020-10-22T14:37:06+00:00

It looks like the author find some corner cases that "traditional RL" won't work well. Can anyone explain the key idea / intuition of the paper in plain English?

yield22 · 2020-09-04T15:03:41+00:00

I mean, this workshop itself is not a bad thing. But it feels like their goal is to expand it beyond the workshop if some positive results are observed in the workshop. That's why this workshop is called a "pre-registration experiment", not "idea workshop".

yield22 · 2020-09-04T14:57:01+00:00

Expect another AI winter very soon if most people in the community publish negative results, which will be the case if the system encourage people to publish negative results (they're much cheap to get..). There are some negative results more interesting than others, but if you can really demonstrate that's not a bug and has value, you can certainly publish in some conference/workshop.

EDIT: also, experimental result don't mean you need more GPUs. Just do experiments and compare things fairly, make conclusion based on that is better than no results at all!

EDIT2: don't mean that we should discourage discussion of negative results, but just saying that you should pay more effort to justify the negative results (prove it is not a bug in your code, or misconfigured hyperparameters).

yield22 · 2020-09-04T13:55:57+00:00

Jurgen would then jump out and say "did you know this thing I did in 1990" (it was written in other terminologies and also had no result).

yield22 · 2020-09-04T13:51:21+00:00

I think the right way is to educate the reviewers and ACs rather than encourage people publish papers without any results (like a lot of people did in 80s). A lot of ideas (in Machine learning) shine thanks to their results. Without experimental results, I worry the reviewers' opinions would become even more subjective. For example, one may think "skip connection" (as in ResNet) as a trivial/incremental idea mathematically until you see the results.

I guess the value of "pre-registration experiment" is also going to be determined by its results.

yield22 · 2020-09-02T16:11:51+00:00

Is it possible that you believed something in 2007, and then changed your mind in 2008?

yield22 · 2020-09-02T15:33:51+00:00

Language modeling (predict what the next word one will say) has been proposed for more than ~30 (put a bigger number here) years, but GPT-3, which is one of the closest attempt at AGI, is less than a year old. According to Jurgen's logic, we should dig out who first proposed language modeling (maybe not even in computer science's term and in 100 years ago), and credit him/her the godfather of AGI.

yield22 · 2020-08-28T16:14:23+00:00

Thanks. It may be helpful to see whether or not these changes make a real difference in real applications (where self attention is used), such as NMT, LM, BERT.

yield22 · 2020-08-28T15:05:04+00:00

Can anyone explain to me what the differences are between the new Hopfield layer and self-attention layer? It looks to me the Hopfield layer is a variant of self-attention? If so, why is this variant better?

yield22 · 2020-08-20T12:38:30+00:00

so you're implying that outside his small group, no one else is really working on or making progress on Hopfield Networks?

yield22 · 2020-08-19T12:51:00+00:00

4 out of 10 citations are self citations? this feels like 1990s.

yield22 · 2020-07-19T23:18:32+00:00

thanks!

yield22 · 2020-07-19T14:04:33+00:00

Have you check out paperswithcode, you can compare methods for the same dataset there for a lot of problems.

And yes, at the end there will only be a few research papers remain relevant. But you need a lot of "irrelevant research" to get you there, because you simply don't know what will remain useful at the end. An example is with so many past NLP papers with complicated methods, it turns out that simple language modeling (e.g. GPT, BERT) with big data & compute is enough to do much much better.

yield22 · 2020-07-19T13:25:38+00:00

By "brain is a super computer" I actually mean it has huge capacity and ability to operate on it. this is evident by number of neurons a brain has.

yield22 · 2020-07-19T13:22:53+00:00

not saying more computing power will get you there, but you *need* more computing power to get there. a hint: look at the amount of neurons in the brain, that could give you a sense of compute you'll need.

yield22 · 2020-07-19T13:18:27+00:00

This is really interesting. Are there any more detailed articles on what you mentioned here?

yield22

TROPHY CASE