Activation functions are essential for neural networks to introduce
non-linearity. A great number of empirical experiments have validated various
activation functions, yet theoretical research on activation functions are
insufficient. In this work, we study the impact of activation functions on the
variance of gradients and propose an approach to normalize activation functions
to keep the variance of the gradient same for all layers so that the neural
network can achieve better convergence. First, we complement the previous work
on the analysis of the variance of gradients where the impact of activation
functions are just considered in an idealized initial state which almost cannot
be preserved during training and obtained a property that good activation
functions should satisfy as possible. Second, we offer an approach to normalize
activation functions and testify its effectiveness on prevalent activation
functions empirically. And by observing experiments, we discover that the speed
of convergence is roughly related to the property we derived in the former
part. We run experiments of our normalized activation functions against common
activation functions. And the result shows our approach consistently
outperforms their unnormalized counterparts. For example, normalized Swish
outperforms vanilla Swish by 1.2% on ResNet50 with CIFAR-100 in terms of top-1
accuracy. Our method improves the performance by simply replacing activation
functions with their normalized ones in both fully-connected networks and
residual networks.

本论文研究了激活函数对神经网络梯度方差的影响，并提出了一种对激活函数进行归一化的方法，以保持所有层的梯度方差相同，从而提高神经网络的收敛性。研究发现，归一化激活函数可显著提高模型性能。