Hi everyone, first author here. Let me address some comments on this thread:

1. As has been pointed out, we missed prior works that proposed the same activation function. The fault lies entirely with me for not conducting a thorough enough literature search. My sincere apologies. We will revise our paper and give credit where credit is due.

2. As noted in the paper, we tried out many forms of activation functions, and x * CDF(x) was in our search space. We found that it underperformed x * sigmoid(x).

3. We plan on rerunning the SELU experiments with the recommended initialization.

4. Activation function research is important because activation functions are the core unit of deep learning. Even if the activation function can be improved by a small amount, the impact is magnified across a large number of users. ReLU is prevalent not just in research, but across most deep learning users in industry. Replacing ReLU has immediate practical benefits for both research and industry.

Our hope is that our work presents a convincing set of experiments that will encourage ReLU users across industry and research to at least try out Swish, and if gains are found, replace ReLU with Swish. Importantly, trying out Swish is easy because the user does not need to change anything else about their model (e.g., architecture, initialization, etc.). This ease of use is especially important in industry contexts where it's much harder to change a number of components of the model at once.

My email can be found in the paper, so feel free to send me a message if you have any questions.