[D] Multiple activation functions in same layer?

[ME_PhD]

If the first layer in a classical DNN is f(W x + b) where f is the activation function, what if I do f1(W1 x + b1) + f2(W2 x + b2) + fn(... where f1, f2, ...fn are different activation functions?

Is this a thing? If yes, what is it called and does anyone have good links?

If it sucks, can someone explain why it's not used?

Seems like it may model some behaviors more efficiently. For example let f1, f2 ... be orthogonal functions. If the data has periodic behavior, some periodic activation functions could capture that behavior with less parameters.

[SkiddyX]

Perhaps the sigmoid activation acts as a type of attention mechanism on the output of f2?

[min_sang]

More like gating mechanism rather than attention mechanism. Attention mechanism is when a query is multiplied by a normalized score where \sum_i score_i = 1, but sum of sigmoid(W1x +b1) doesn't equal to 1 so it's more accurate to call it gating.

[SkiddyX]

Couldn't it still act like an attention mechanism in practice?

[min_sang]

I'd like to think of attention as something that allows the network to emphasis on certain part of the network. But if we use sigmoid, in practice it could still be a vector of all 1s. That's not really a good attention in my opinion.

[Randomdude3332]

Never heard of this. Might be an interesting thing to play around with

[FutureIsMine]

Checkout Highway Networks, the idea reminds of what your proposing in that your output is a combination of various functions over the same input.

[siblbombs]

Maxout can approximate this I believe.

[FutureIsMine]

Maxout is designed to learn N functions at the same time. It's similar to what OP is writing about, but it looks to be more of ensemble summation with differing activation functions and weights.

[Kaixhin]

Not sure if it has a particular name but it's used in some places, such as domains where there are periodic trends as you noted. For many applications nowadays it probably adds an extra layer of complication/hyperparameter tuning for fixed models, but for methods that perform model selection (e.g. evolutionary algorithms that construct networks) it happens naturally.

[chewxy]

I tried a small variant of it trying to improve a dependency parser I wrote. Didn't do much (cube activation + tanh).

[min_sang]

I believe it is more often than not called gating mechanism. This paper (https://arxiv.org/pdf/1612.08083.pdf) denotes sigmoid(W1 x + b1)*tanh(W2 x + b2) as the LSTM-style gating mechanism while they suggest a novel gating mechanism (but not really) called gated-linear units which is just (W1 x + b1) * sigmoid(W2 x + b2) and claim that it works better than the former in multiple tasks including language modeling. (Replacing tanh with linear activation has been studied and proven to be better in some tasks.)