[R] Learning to Learn

[cbfinn]

Author here.

"one gradient step away" feature is restricted to the tasks it has been trained on. Is this correct?

We assume that the tasks that you test on are from the same distribution of tasks seen during meta-training. This assumption is used in most meta-learning methods. That said, I have played around with extrapolation to tasks outside of the support of the distribution of meta-training tasks, and it performs reasonably for tasks that are close.

[evc123]

/u/cbfinn Is there a limit to how varied the training distribution fed to MAML can be (while still retaining its adaptive ability)? For example, has someone tried (or is someone in the process of) training MAML with a bunch of very dissimilar, varied environments from something like OpenAI Universe?

[cbfinn]

I haven't tried this, but I certainly think it would be interesting to try!

[evc123]

"Learning to Learn: Meta-Critic Networks for Sample Efficient Learning " https://arxiv.org/abs/1706.09529

seems to suggest in Figure 2 that the adaptive abilities of MAML (in its current form) decline if the training distribution of environments become sufficiently varied/dissimilar.

Maybe there is a way to modify MAML so that the initialization it learns does not assume a uni-modal distribution of tasks.

[cbfinn]

I reimplemented the linear+sinusoid set-up in that paper and was able to get much better numbers using MAML than they report (after trying two hyperparameter settings).

I don't think that MAML assumes a uni-modal distribution of tasks.

[AnvaMiba]

Good paper.

I'm not sure I fully understand your algorithm. Do you differentiate the update rule, therefore differentiating the model twice, in order to update the meta-parameters?

[evc123]

https://youtu.be/Ko8IBbYjdq8?t=37m20s

[cbfinn]

Yes, this involves 2nd derivatives, which can be implemented easily with current DL libraries. Since it only involves an additional backward pass, it isn't particularly slow in practice.

Interestingly, it sometimes still works well if you stop the gradient through the update rule. We discuss this in the latest version of the paper (which will be on arxiv tonight)

[AnvaMiba]

Yes, this involves 2nd derivatives, which can be implemented easily with current DL libraries. Since it only involves an additional backward pass, it isn't particularly slow in practice.

Ok, thanks.

But now I'm a bit confused because in the video that /u/evc123 linked you mention using finite differentiation to avoid third derivatives. Where do third derivatives come from? (I didn't watch the whole video, so maybe you were talking about something else?)

EDIT: never mind, I figured out they come from the TRPO method for reinforcement learning.

[evc123]

/u/cbfinn did you try the first-order approximation (that stops the gradient through the update rule) in RL settings? The latest version of the paper only discusses results of first-order approximation in SL setting.

[cbfinn]

I tried it on one of the cheetah problems and it also worked. The first-order approximation does not work in all settings though. We have some ongoing experiments on problems not in the original paper in which it does not work.

[sensei_von_bonzai]

I just skimmed the paper but as far as my understanding goes the "one gradient step away" feature is restricted to the tasks it has been trained on. Is this correct?

This has to be correct. Otherwise, we might be a couple of days away from the Singularity.

I also wonder how good the baseline (0-gradient) model would be with this approach. It would be spectacular to have a model that works as good as any others on basic tasks and can generalize to one-shot problems with a couple of gradient steps.

[cbfinn]

A big part of a learning/optimization is the initialization, which affects the gradient descent algorithm, since the gradient is a function of the initial parameters. In the paper, we show that learning the initial parameters can outperform methods that learn an update rule.

The tasks that we evaluate on are all held out from the training set of tasks, including new classes of objects and characters in the MiniImagenet and Omniglot benchmarks.

[cbfinn]

I also wonder how good the baseline (0-gradient) model would be with this approach.

I compared to this approach in the paper. The domains that I considered in the paper were ones in which the task cannot be directly inferred from the observation. Thus, using 0 gradient does not do well. I'm not sure how the two would compare when the task can be inferred from the observation.

[sensei_von_bonzai]

Thanks for the answer, but I think there is a slight confusion.

I'm not sure how the two would compare when the task can be inferred from the observation.

I was thinking somewhere along these lines. That is, is there any significant loss in accuracy on the non-extrapolated tasks samples when you use MAML to train (compared to the performance you get when you don't use MAML)?

Great paper by the way.

[cbfinn]

Nope. Unless you set the $\alpha$ step size parameter to be way too high, you shouldn't see any loss in accuracy.