The choice of batch-size in a stochastic optimization algorithm plays a
substantial role for both optimization and generalization. Increasing the
batch-size used typically improves optimization but degrades generalization. To
address the problem of improving generalization while maintaining optimal
convergence in large-batch training, we propose to add covariance noise to the
gradients. We demonstrate that the learning performance of our method is more
accurately captured by the structure of the covariance matrix of the noise
rather than by the variance of gradients. Moreover, over the convex-quadratic,
we prove in theory that it can be characterized by the Frobenius norm of the
noise matrix. Our empirical studies with standard deep learning
model-architectures and datasets shows that our method not only improves
generalization performance in large-batch training, but furthermore, does so in
a way where the optimization performance remains desirable and the training
duration is not elongated.

本研究提出一种改善大批量训练的泛化性能并维持最优收敛的方法，即在梯度中添加协方差噪声，通过对噪声矩阵的对数 Frobenius 范数进行特征提取获得更精确的学习效果表现。实证研究表明该方法不仅可以提高大批量训练的泛化性能，而且不会影响优化性能以及训练时间。

大批次随机梯度下降与结构协方差噪声的实证研究

An Empirical Study of Large-Batch Stochastic Gradient Descent with  Structured Covariance Noise

Stochastic Gradient Descent (SGD) based training of neural networks with a
large learning rate or a small batch-size typically ends in well-generalizing,
flat regions of the weight space, as indicated by small eigenvalues of the
Hessian of the training loss. However, the curvature along the SGD trajectory
is poorly understood. An empirical investigation shows that initially SGD
visits increasingly sharp regions, reaching a maximum sharpness determined by
both the learning rate and the batch-size of SGD. When studying the SGD
dynamics in relation to the sharpest directions in this initial phase, we find
that the SGD step is large compared to the curvature and commonly fails to
minimize the loss along the sharpest directions. Furthermore, using a reduced
learning rate along these directions can improve training speed while leading
to both sharper and better generalizing solutions compared to vanilla SGD. In
summary, our analysis of the dynamics of SGD in the subspace of the sharpest
directions shows that they influence the regions that SGD steers to (where
larger learning rate or smaller batch size result in wider regions visited),
the overall training speed, and the generalization ability of the final model.

使用较小的学习率和 SGD 最陡峭的方向进行训练可以提高模型的训练速度和泛化能力，而较大的学习率或较小的批量大小将导致 SGD 进入更宽的区域。