In this work we propose a new non-monotonic activation function: the modulus.
The majority of the reported research on nonlinearities is focused on monotonic
functions. We empirically demonstrate how by using the modulus activation
function on computer vision tasks the models generalize better than with other
nonlinearities - up to a 15% accuracy increase in CIFAR100 and 4% in CIFAR10,
relative to the best of the benchmark activations tested. With the proposed
activation function the vanishing gradient and dying neurons problems
disappear, because the derivative of the activation function is always 1 or -1.
The simplicity of the proposed function and its derivative make this solution
specially suitable for TinyML and hardware applications.

本文提出了一种新的非单调激活函数 —— 模函数，在计算机视觉任务中使用该函数可以使模型相对于其他非线性函数具有更好的泛化能力，同时还能解决梯度消失和神经元失活问题，这种简单易用的函数对于 TinyML 和硬件应用尤其适用。