Recent research has found that the activation function (AF) selected for
adding non-linearity into the output can have a big impact on how effectively
deep learning networks perform. Developing activation functions that can adapt
simultaneously with learning is a need of time. Researchers recently started
developing activation functions that can be trained throughout the learning
process, known as trainable, or adaptive activation functions (AAF). Research
on AAF that enhance the outcomes is still in its early stages. In this paper, a
novel activation function 'ErfReLU' has been developed based on the erf
function and ReLU. This function exploits the ReLU and the error function (erf)
to its advantage. State of art activation functions like Sigmoid, ReLU, Tanh,
and their properties have been briefly explained. Adaptive activation functions
like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and
Serf have also been described. Lastly, performance analysis of 9 trainable
activation functions along with the proposed one namely Tanhsoft1, Tanhsoft2,
Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf has been shown by
applying these activation functions in MobileNet, VGG16, and ResNet models on
CIFAR-10, MNIST, and FMNIST benchmark datasets.

本文介绍了一种基于 Erf 函数和 ReLU 的新型激活函数 'ErfReLU'，并比较了其与其他 9 种可训练激活函数在 CIFAR-10，MNIST 和 FMNIST 基准数据集上应用于 MobileNet，VGG16，ResNet 模型的性能分析。