Sharpness-Aware Minimization (SAM) is most known for achieving state-of
the-art performances on natural image and language tasks. However, its most
pronounced improvements (of tens of percent) is rather in the presence of label
noise. Understanding SAM's label noise robustness requires a departure from
characterizing the robustness of minimas lying in "flatter" regions of the loss
landscape. In particular, the peak performance under label noise occurs with
early stopping, far before the loss converges. We decompose SAM's robustness
into two effects: one induced by changes to the logit term and the other
induced by changes to the network Jacobian. The first can be observed in linear
logistic regression where SAM provably up-weights the gradient contribution
from clean examples. Although this explicit up-weighting is also observable in
neural networks, when we intervene and modify SAM to remove this effect,
surprisingly, we see no visible degradation in performance. We infer that SAM's
effect in deeper networks is instead explained entirely by the effect SAM has
on the network Jacobian. We theoretically derive the implicit regularization
induced by this Jacobian effect in two layer linear networks. Motivated by our
analysis, we see that cheaper alternatives to SAM that explicitly induce these
regularization effects largely recover the benefits in deep networks trained on
real-world datasets.

Sharpness-Aware Minimization 对标签噪声鲁棒性的理解需要考虑到对误差曲面中 “较平坦” 区域最小值的鲁棒性以及提前停止对峰值性能的影响。通过对 logit 项和网络 Jacobian 的改变进行分解，我们推断深层网络中的 SAM 效果完全由其对网络 Jacobian 的影响解释，进一步推导了这种 Jacobian 效果在两层线性网络中引发的隐式正则化效应。受我们分析的启发，我们发现在真实数据集上训练的深度网络中，明确引入这些正则化效应的成本更低的 SAM 替代方法在很大程度上恢复了优势。