Recently, Sharpness-Aware Minimization (SAM) algorithm has shown
state-of-the-art generalization abilities in vision tasks. It demonstrates that
flat minima tend to imply better generalization abilities. However, it has some
difficulty implying SAM to some natural language tasks, especially to models
with drastic gradient changes, such as RNNs. In this work, we analyze the
relation between the flatness of the local minimum and its generalization
ability from a novel and straightforward theoretical perspective. We propose
that the shift of the training and test distributions can be equivalently seen
as a virtual parameter corruption or perturbation, which can explain why flat
minima that are robust against parameter corruptions or perturbations have
better generalization performances. On its basis, we propose a
Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM)
algorithm to help to learn algorithms find flat minima that generalize better.
Results in various language benchmarks validate the effectiveness of the
proposed GA-SAM algorithm on natural language tasks.

本文从理论角度分析了局部最小值的平坦程度与模型泛化性能的关系，提出了基于梯度强度的自适应 Sharpness-Aware Minimization 算法（GA-SAM）来发现更具鲁棒性的平坦局部最小值，其在自然语言任务上的表现令人满意。