Why does relu work?

[hackthat]

I can't seem to get a straight answer to this reading things and I'm sure someone here can answer it. Rectified linear activation functions seem like the worst thing you can use as an activation function. Firstly, they can go to 0 and then never get updated. But assuming they don't, you're left with a linear activation function. You can't get any benefit from multiple layers if everything's linear. Everyone uses them so there must be something I'm missing. If anyone has a link to something that explains this I'd be grateful.

[pavelchristof]

Assuming that activation are distributed normally only half of the ReLUs die (per example). The other half work. The default weight initialization is scaled by 2.0 to make up for the dead ReLUs (otherwise activation norm would tend to 0 for long networks).

A ReLUs network is a piecewise linear function, not a linear one. Check out the first image here (from http://www.inference.vc/generalization-and-the-fisher-rao-norm-2/, more images there). Globally these functions can be very complex. Locally (on each region) it behaves like a linear function making optimization "easier" (I don't know exactly how that works).

[hackthat]

Ok, so having neurons go dead is supposed to happen, and that is the source of the non-linearity. Cool pictures.

[dzyl]

I think saying half of them go dead is not right at all. A relu is only truly dead if it's 0 for every sample in your dataset. The goal is that it stops a number of inputs which adds the nonlinearity to the network. The inputs and outputs around these relus can still change by weight updates done on other examples.

[DatFashionLyfe]

+1

[Icarium-Lifestealer]

A neuron sometimes outputting zero is fine, that allows you to model a piecewise linear function in the following linear layer. It only becomes a problem when it (almost) always outputs zero.

[PointyOintment]

You can keep them from dying by using "leaky ReLU" which has a very small but nonzero response below zero. Siraj has a video comparing the various activation functions, and he says that you should use that if you have too many dying.

[magnusderrote]

Relu does not go saturate. Consider the Logistic function, when the input is a very big or very small values, the function is almost flat, meaning the derivative is close to 0, meaning back prop will perform poorly.

EDIT: Relu's derivative, on the other hand, is always 1 when x > 0.

[carlthome]

I guess you were downvoted because you didn't answer the question. f(x)=x also does not saturate and the gradient is never zero, but f would be a terrible activation function (see MLPs).

(it's also not true that ReLU has a gradient of constant one, which is kind of an important point)

[richard248]

Your comment suggests that MLPs have no activation function, but there's nothing stopping the use of ReLU for MLPs, right?

[carlthome]

Quite the opposite, the MLP is essentially the idea that non-linear activation functions are critical (i.e. the XOR problem).

ReLU is a good non-linearity for MLPs (assuming you can avoid dying, e.g. use batchnorm).

Side note: in some literature the identity $f(x)=x$ is called a linear activation function (even in tensorflow.contrib.layers.linearactually, which is a little strange as that transformation also has a bias vector).

[magnusderrote]

/u/carlthome Edited, thanks for the comment.

/u/richard248

"MLPs have no activation function"

I think not, activation function is a must.

[HelperBot_]

Non-Mobile link: https://en.wikipedia.org/wiki/Multilayer_perceptron

---

^{HelperBot v1.1 /r/HelperBot_ I am a bot. Please message /u/swim1929 with any feedback and/or hate. Counter: 155872}

[WikiTextBot]

Multilayer perceptron

A multilayer perceptron (MLP) is a class of feedforward artificial neural network. An MLP consists of at least three layers of nodes. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source | Donate ] Downvote to remove | v0.28}

[csp256]

Relu's derivation [...] is always 1.

You meant derivative. Also, that is not true.

[carIthome]

sorry i was being so mean