Optimization techniques are of great importance to effectively and
efficiently train a deep neural network (DNN). It has been shown that using the
first and second order statistics (e.g., mean and variance) to perform Z-score
standardization on network activations or weight vectors, such as batch
normalization (BN) and weight standardization (WS), can improve the training
performance. Different from these existing methods that mostly operate on
activations or weights, we present a new optimization technique, namely
gradient centralization (GC), which operates directly on gradients by
centralizing the gradient vectors to have zero mean. GC can be viewed as a
projected gradient descent method with a constrained loss function. We show
that GC can regularize both the weight space and output feature space so that
it can boost the generalization performance of DNNs. Moreover, GC improves the
Lipschitzness of the loss function and its gradient so that the training
process becomes more efficient and stable. GC is very simple to implement and
can be easily embedded into existing gradient based DNN optimizers with only
one line of code. It can also be directly used to fine-tune the pre-trained
DNNs. Our experiments on various applications, including general image
classification, fine-grained image classification, detection and segmentation,
demonstrate that GC can consistently improve the performance of DNN learning.
The code of GC can be found at
this https URL

本文提出了一种新型优化方案 —— 梯度集中（Gradient Centralization，GC），通过将梯度向量集中到零均值来优化神经网络训练，这可以被视为一种具有约束损失函数的投影梯度下降法，实验证明它能够显著优化深度神经网络的泛化能力、提高训练效率和稳定性。