In this paper, we first identify activation shift, a simple but remarkable
phenomenon in a neural network in which the preactivation value of a neuron has
non-zero mean that depends on the angle between the weight vector of the neuron
and the mean of the activation vector in the previous layer. We then propose
linearly constrained weights (LCW) to reduce the activation shift in both fully
connected and convolutional layers. The impact of reducing the activation shift
in a neural network is studied from the perspective of how the variance of
variables in the network changes through layer operations in both forward and
backward chains. We also discuss its relationship to the vanishing gradient
problem. Experimental results show that LCW enables a deep feedforward network
with sigmoid activation functions to be trained efficiently by resolving the
vanishing gradient problem. Moreover, combined with batch normalization, LCW
improves generalization performance of both feedforward and convolutional
networks.

在本文中，我们首先识别到神经网络中的激活偏移现象，通过线性约束权重（LCW）来减小激活偏移，研究了减小激活偏移对神经网络中变量方差的影响，并讨论其与梯度消失问题的关系。实验结果表明，LCW 能够通过解决梯度消失问题有效地训练具有 sigmoid 激活函数的深度前馈网络，并结合批标准化能够改善前馈和卷积网络的泛化性能。