Activation functions are essential for neural networks to introduce
non-linearity. A great number of empirical experiments have validated various
activation functions, yet theoretical research on activation functions are
insufficient. In this work, we study the impact of activation functions on the
variance of gradients and propose an approach to normalize activation functions
to keep the variance of the gradient same for all layers so that the neural
network can achieve better convergence. First, we complement the previous work
on the analysis of the variance of gradients where the impact of activation
functions are just considered in an idealized initial state which almost cannot
be preserved during training and obtained a property that good activation
functions should satisfy as possible. Second, we offer an approach to normalize
activation functions and testify its effectiveness on prevalent activation
functions empirically. And by observing experiments, we discover that the speed
of convergence is roughly related to the property we derived in the former
part. We run experiments of our normalized activation functions against common
activation functions. And the result shows our approach consistently
outperforms their unnormalized counterparts. For example, normalized Swish
outperforms vanilla Swish by 1.2% on ResNet50 with CIFAR-100 in terms of top-1
accuracy. Our method improves the performance by simply replacing activation
functions with their normalized ones in both fully-connected networks and
residual networks.

本论文研究了激活函数对神经网络梯度方差的影响，并提出了一种对激活函数进行归一化的方法，以保持所有层的梯度方差相同，从而提高神经网络的收敛性。研究发现，归一化激活函数可显著提高模型性能。

标准化激活函数：迈向更好收敛

Normalized Activation Function: Toward Better Convergence

In machine learning, a question of great interest is understanding what
examples are challenging for a model to classify. Identifying atypical examples
ensures the safe deployment of models, isolates samples that require further
human inspection and provides interpretability into model behavior. In this
work, we propose Variance of Gradients (VoG) as a valuable and efficient metric
to rank data by difficulty and to surface a tractable subset of the most
challenging examples for human-in-the-loop auditing. We show that data points
with high VoG scores are far more difficult for the model to learn and
over-index on corrupted or memorized examples. Further, restricting the
evaluation to the test set instances with the lowest VoG improves the model's
generalization performance. Finally, we show that VoG is a valuable and
efficient ranking for out-of-distribution detection.

该论文提出了一种名为 VoG 的评估指标，可以通过评估模型求解困难样本，从而实现模型的可解释性和泛化性能，并在数据样本的噪声和记忆样本方面得到应用。