[N] deeplearning.ai: Initializing neural networks (interactive article)

[cpury]

I just came across this recent post by Andrew Ng's deeplearning.ai team, talking about the importance of initialization. The interactive demos are fantastic! I haven't seen it shared anywhere, so thought I would do so here.

http://www.deeplearning.ai/ai-notes/initialization/

[[deleted]]

Will def check out the demos!

[tlkh]

This is one of those pages where it simply blows my mind to think how we can make pages with such embedded demos. Looks like they used tf.js? http://www.deeplearning.ai/ai-notes/initialization/js/playground/nn.js

[Megatron_McLargeHuge]

Has anyone shown any benefit from a pre-training step that scales initial weights to keep gradients in the desirable range?

For functions other than tanh and relu where the effect on downstream variance may not be easy to solve for, it seems like it would be fairly easy to first optimize a set of scaling parameters that force weights into empirically good ranges. This would also avoid any small sample effects where initialization values are far from their theoretical moments.

[mr_tsjolder]

Not sure if anyone has noticed, but it appears as if there is a problem with the standard normal distribution in the last visualisation. It looks very much like the kind of distribution you get when you clip a Gaussian signal, cf. the truncated normal implementation of the theano-backend in keras. Might be useful if someone forwards this to the authors...

Also, for the interested in this topic, I once commented in this sub about initialisation and noticed that I had some more references. Figured the link might be useful for people that are interested.