The choice of activation functions in deep networks has a significant effect
on the training dynamics and task performance. Currently, the most successful
and widely-used activation function is the Rectified Linear Unit (ReLU).
Although various hand-designed alternatives to ReLU have been proposed, none
have managed to replace it due to inconsistent gains. In this work, we propose
to leverage automatic search techniques to discover new activation functions.
Using a combination of exhaustive and reinforcement learning-based search, we
discover multiple novel activation functions. We verify the effectiveness of
the searches by conducting an empirical evaluation with the best discovered
activation function. Our experiments show that the best discovered activation
function, $f(x) = x \cdot \text{sigmoid}(\beta x)$, which we name Swish, tends
to work better than ReLU on deeper models across a number of challenging
datasets. For example, simply replacing ReLUs with Swish units improves top-1
classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for
Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it
easy for practitioners to replace ReLUs with Swish units in any neural network.

本文提出利用自动搜索技术发现新的激活函数。通过详尽和强化学习的结合搜索，发现了多个新型激活函数，其中最佳的发现激活函数（称为 Swish）在许多困难数据集上比 ReLU 更有效。