Sharpness-Aware Minimization (SAM) is a recent training method that relies on
worst-case weight perturbations which significantly improves generalization in
various settings. We argue that the existing justifications for the success of
SAM which are based on a PAC-Bayes generalization bound and the idea of
convergence to flat minima are incomplete. Moreover, there are no explanations
for the success of using $m$-sharpness in SAM which has been shown as essential
for generalization. To better understand this aspect of SAM, we theoretically
analyze its implicit bias for diagonal linear networks. We prove that SAM
always chooses a solution that enjoys better generalization properties than
standard gradient descent for a certain class of problems, and this effect is
amplified by using $m$-sharpness. We further study the properties of the
implicit bias on non-linear networks empirically, where we show that
fine-tuning a standard model with SAM can lead to significant generalization
improvements. Finally, we provide convergence results of SAM for non-convex
objectives when used with stochastic gradients. We illustrate these results
empirically for deep networks and discuss their relation to the generalization
behavior of SAM. The code of our experiments is available at
this https URL

Sharpness-Aware Minimization (SAM) relies on worst-case weight perturbations to improve generalization; we provide a more complete theoretical framework for SAM's success, analyze its implicit bias on diagonal linear networks and empirically on fine-tuning non-linear networks, and provide convergence results for non-convex objectives when used with stochastic gradients.

探索锐度感知最小化理解

Towards Understanding Sharpness-Aware Minimization

We provide a detailed asymptotic study of gradient flow trajectories and
their implicit optimization bias when minimizing the exponential loss over
"diagonal linear networks". This is the simplest model displaying a transition
between "kernel" and non-kernel ("rich" or "active") regimes. We show how the
transition is controlled by the relationship between the initialization scale
and how accurately we minimize the training loss. Our results indicate that
some limit behaviors of gradient descent only kick in at ridiculous training
accuracies (well beyond $10^{-100}$). Moreover, the implicit bias at reasonable
initialization scales and training accuracies is more complex and not captured
by these limits.

研究了在 “对角线线性网络” 上最小化指数损失的梯度流轨迹及其隐式优化偏差的详细渐近研究，揭示了 “核” 与非 “核”（“丰富” 或 “活跃”）状态之间的转换如何受初始缩放和最小化训练损失的精度之间的关系控制，结果表明，梯度下降的某些极限行为仅在荒谬的训练精度（远远低于 $10^{-100}$）时才能发挥作用。此外，在合理的初始化尺度和训练精度下，隐式偏差更为复杂，超出了这些极限的范畴。