Sharpness-Aware Minimization (SAM) has been instrumental in improving deep
neural network training by minimizing both training loss and loss sharpness.
Despite the practical success, the mechanisms behind SAM's generalization
enhancements remain elusive, limiting its progress in deep learning
optimization. In this work, we investigate SAM's core components for
generalization improvement and introduce "Friendly-SAM" (F-SAM) to further
enhance SAM's generalization. Our investigation reveals the key role of
batch-specific stochastic gradient noise within the adversarial perturbation,
i.e., the current minibatch gradient, which significantly influences SAM's
generalization performance. By decomposing the adversarial perturbation in SAM
into full gradient and stochastic gradient noise components, we discover that
relying solely on the full gradient component degrades generalization while
excluding it leads to improved performance. The possible reason lies in the
full gradient component's increase in sharpness loss for the entire dataset,
creating inconsistencies with the subsequent sharpness minimization step solely
on the current minibatch data. Inspired by these insights, F-SAM aims to
mitigate the negative effects of the full gradient component. It removes the
full gradient estimated by an exponentially moving average (EMA) of historical
stochastic gradients, and then leverages stochastic gradient noise for improved
generalization. Moreover, we provide theoretical validation for the EMA
approximation and prove the convergence of F-SAM on non-convex problems.
Extensive experiments demonstrate the superior generalization performance and
robustness of F-SAM over vanilla SAM. Code is available at
this https URL

SAM 的关键组件有助于改进模型泛化性能，而 F-SAM 则通过移除全梯度成分并利用随机梯度噪声来进一步提高模型的泛化性能。

友好的锐度感知最小化

Friendly Sharpness-Aware Minimization

It is well-known that stochastic gradient noise (SGN) acts as implicit
regularization for deep learning and is essentially important for both
optimization and generalization of deep networks. Some works attempted to
artificially simulate SGN by injecting random noise to improve deep learning.
However, it turned out that the injected simple random noise cannot work as
well as SGN, which is anisotropic and parameter-dependent. For simulating SGN
at low computational costs and without changing the learning rate or batch
size, we propose the Positive-Negative Momentum (PNM) approach that is a
powerful alternative to conventional Momentum in classic optimizers. The
introduced PNM method maintains two approximate independent momentum terms.
Then, we can control the magnitude of SGN explicitly by adjusting the momentum
difference. We theoretically prove the convergence guarantee and the
generalization advantage of PNM over Stochastic Gradient Descent (SGD). By
incorporating PNM into the two conventional optimizers, SGD with Momentum and
Adam, our extensive experiments empirically verified the significant advantage
of the PNM-based variants over the corresponding conventional Momentum-based
optimizers.

提出了一种 Positive-Negative Momentum (PNM) 方法来模拟随机梯度噪声 (SGN) 并代替传统的动量方法，通过将两个近似独立的动量项的差异调整来显式控制 SGN 的大小，理论上证明了 PNM 相比于 SGD 的收敛保证和泛化优势，并在将 PNM 应用于 SGD 和 Adam 两个传统优化器后，实验证明了 PNM 优化器的巨大优势。

正负动量：利用随机梯度噪声来改善泛化

Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to  Improve Generalization

We study the effect of the stochastic gradient noise on the training of
generative adversarial networks (GANs) and show that it can prevent the
convergence of standard game optimization methods, while the batch version
converges. We address this issue with a novel stochastic variance-reduced
extragradient (SVRE) optimization algorithm, which for a large class of games
improves upon the previous convergence rates proposed in the literature. We
observe empirically that SVRE performs similarly to a batch method on MNIST
while being computationally cheaper, and that SVRE yields more stable GAN
training on standard datasets.

我们研究了随机梯度噪声对生成对抗网络（GAN）训练的影响，并表明它可以防止标准游戏优化方法的收敛，而批量版本收敛。我们提出了一种新颖的随机方差减小外推（SVRE）优化算法，它可以为大多数游戏类别提高文献提出的收敛速度。我们在 MNIST 上经验性地观察到 SVRE 在计算上比批处理方法更便宜，并且 SVRE 在标准数据集上产生更稳定的 GAN 训练。