Sharpness-Aware Minimization (SAM) has emerged as a promising approach for
effectively reducing the generalization error. However, SAM incurs twice the
computational cost compared to base optimizer (e.g., SGD). We propose
Asymptotic Unbiased Sampling with respect to iterations to accelerate SAM
(AUSAM), which maintains the model's generalization capacity while
significantly enhancing computational efficiency. Concretely, we
probabilistically sample a subset of data points beneficial for SAM
optimization based on a theoretically guaranteed criterion, i.e., the Gradient
Norm of each Sample (GNS). We further approximate the GNS by the difference in
loss values before and after perturbation in SAM. As a plug-and-play,
architecture-agnostic method, our approach consistently accelerates SAM across
a range of tasks and networks, i.e., classification, human pose estimation and
network quantization. On CIFAR10/100 and Tiny-ImageNet, AUSAM achieves results
comparable to SAM while providing a speedup of over 70%. Compared to recent
dynamic data pruning methods, AUSAM is better suited for SAM and excels in
maintaining performance. Additionally, AUSAM accelerates optimization in human
pose estimation and model quantization without sacrificing performance,
demonstrating its broad practicality.

使用渐近无偏采样技术加速 Sharpness-Aware Minimization (SAM) 优化算法，提供超过 70% 的加速效果，同时保持性能。