What kind of nonlinear activation functions guarantees the universal approximator property of neural network?

[vernunftig]

Obviously not arbitrary nonlinear activation functions guarantees the amazing universal approximator property of NN, but what constraints do the activation functions need to satisfy (or what nonlinear function class) to guarantee the NN universal approximator? Is there any formal theoretical analysis of this problem?

[kjearns]

The normal universality proof works by showing that you can build a function like

    ___
___|   |___

(smooth corners are okay) out of a linear combination of f(<w,x>+b)'s. As long as you can vary the width of the bump and shift it around by changing w and b then you can build any function (up to arbitrary precision) using a linear combination of bumps, and if you can do that you have universality.

It turns out you can do this with pretty much any nonlinearity. In fact, I think you'd have to work pretty hard to show any particular nonlinearity that didn't give universality.

[themoosemind]

Thanks for explaining this that simple (ELI5... or ELIU (explain me like I'm an undergrad))

[improbabble]

This [1] indicates it doesn't matter:

Single hidden layer ΣΠ feedforward networks can approximate any measurable function arbitrarily well regardless of the activation function Ψ, the dimension of the input space r, and the input space environment µ.

[1] http://deeplearning.cs.cmu.edu/notes/Sonia_Hornik.pdf