Sharpness-aware Minimization (SAM) has been proposed recently to improve
model generalization ability. However, SAM calculates the gradient twice in
each optimization step, thereby doubling the computation costs compared to
stochastic gradient descent (SGD). In this paper, we propose a simple yet
efficient sampling method to significantly accelerate SAM. Concretely, we
discover that the gradient of SAM is a combination of the gradient of SGD and
the Projection of the Second-order gradient matrix onto the First-order
gradient (PSF). PSF exhibits a gradually increasing frequency of change during
the training process. To leverage this observation, we propose an adaptive
sampling method based on the variation of PSF, and we reuse the sampled PSF for
non-sampling iterations. Extensive empirical results illustrate that the
proposed method achieved state-of-the-art accuracies comparable to SAM on
diverse network architectures.

通过采用自适应采样方法基于 PSF 变化的原则，本文提出了一种简单且高效的采样方法，以显著加速模型优化过程中的计算速度，并在广泛的网络架构上实现了与 SAM 相当的最新准确度。