Activation functions play a pivotal role in determining the training dynamics
and neural network performance. The widely adopted activation function ReLU
despite being simple and effective has few disadvantages including the Dying
ReLU problem. In order to tackle such problems, we propose a novel activation
function called Serf which is self-regularized and nonmonotonic in nature. Like
Mish, Serf also belongs to the Swish family of functions. Based on several
experiments on computer vision (image classification and object detection) and
natural language processing (machine translation, sentiment classification and
multimodal entailment) tasks with different state-of-the-art architectures, it
is observed that Serf vastly outperforms ReLU (baseline) and other activation
functions including both Swish and Mish, with a markedly bigger margin on
deeper architectures. Ablation studies further demonstrate that Serf based
architectures perform better than those of Swish and Mish in varying scenarios,
validating the effectiveness and compatibility of Serf with varying depth,
complexity, optimizers, learning rates, batch sizes, initializers and dropout
rates. Finally, we investigate the mathematical relation between Swish and
Serf, thereby showing the impact of preconditioner function ingrained in the
first derivative of Serf which provides a regularization effect making
gradients smoother and optimization faster.

提出了一种自我调节且非单调的激活函数 Serf，该函数可大幅提高神经网络的性能，在计算机视觉等多种任务中表现优异，深层网络中效果更为显著。与其他常用激活函数相比，Serf 更兼容各种深度、复杂度、优化器、学习率、批量大小、初始化器和节点丢弃率等方面的设置，理论上表现修正了基于 Swish 函数的 Mish 函数的先决条件函数可充当的有效性，通过数学关系证明了 Swish 与 Serf 之间的关系，为深度学习以及神经网络的未来发展提供新思路。