We characterize the learning dynamics of stochastic gradient descent (SGD) when continuous symmetry exists in the loss function, where the divergence between SGD and gradient descent is dramatic. We show that depending on how the symmetry affects the learning dynamics, we can divide a family of symmetry into two classes. For one class of symmetry, SGD naturally converges to solutions that have a balanced and aligned gradient noise. For the other class of symmetry, SGD will almost always diverge. Then, we show that our result remains applicable and can help us understand the training dynamics even when the symmetry is not present in the loss function. Our main result is universal in the sense that it only depends on the existence of the symmetry and is independent of the details of the loss function. We demonstrate that the proposed theory offers an explanation of progressive sharpening and flattening and can be applied to common practical problems such as representation normalization, matrix factorization, and the use of warmup.

当存在连续对称性时，我们表征了随机梯度下降法（SGD）的学习动力学，它与梯度下降法之间的差异是显著的。我们将对称性对学习动力学的影响分为两类：其中一类对称性，SGD自然地收敛于具有平衡和对齐梯度噪声的解；另一类对称性下，SGD几乎总是发散。然后，我们证明了即使丢失函数中不存在对称性，我们的结果仍然适用并可以帮助我们理解训练动力学。我们的主要结果是普适的，仅依赖于对称性的存在，而与丢失函数的细节无关。我们证明了提出的理论解释了渐进锐化和平坦化，并可应用于常见的实际问题，如表示规范化，矩阵分解和预热的使用。

梯度噪声的隐含偏差：对称性视角