Recently, there has been a surge in interest in developing optimization
algorithms for overparameterized models as achieving generalization is believed
to require algorithms with suitable biases. This interest centers on minimizing
sharpness of the original loss function; the Sharpness-Aware Minimization (SAM)
algorithm has proven effective. However, most literature only considers a few
sharpness measures, such as the maximum eigenvalue or trace of the training
loss Hessian, which may not yield meaningful insights for non-convex
optimization scenarios like neural networks. Additionally, many sharpness
measures are sensitive to parameter invariances in neural networks, magnifying
significantly under rescaling parameters. Motivated by these challenges, we
introduce a new class of sharpness measures in this paper, leading to new
sharpness-aware objective functions. We prove that these measures are
\textit{universally expressive}, allowing any function of the training loss
Hessian matrix to be represented by appropriate hyperparameters. Furthermore,
we show that the proposed objective functions explicitly bias towards
minimizing their corresponding sharpness measures, and how they allow
meaningful applications to models with parameter invariances (such as
scale-invariances). Finally, as instances of our proposed general framework, we
present \textit{Frob-SAM} and \textit{Det-SAM}, which are specifically designed
to minimize the Frobenius norm and the determinant of the Hessian of the
training loss, respectively. We also demonstrate the advantages of our general
framework through extensive experiments.

最近，对于过参数化模型开发优化算法的兴趣不断增加，因为普遍认为实现泛化需要具有适当偏差的算法。本文针对仅考虑少数尖锐度测量值的文献提出了新的尖锐度测量方法，并证明了这些测量方法的普遍表达性，以及它们如何对模型的参数不变性有着显著的影响。此外，本文还提出了 Frob-SAM 和 Det-SAM 两种特定设计的具体优化框架，并通过大量实验证明了该框架的优势。

一个通用的尖锐感知最小化算法类

A Universal Class of Sharpness-Aware Minimization Algorithms

In this paper, we study the sharpness of a deep learning (DL) loss landscape
around local minima in order to reveal systematic mechanisms underlying the
generalization abilities of DL models. Our analysis is performed across varying
network and optimizer hyper-parameters, and involves a rich family of different
sharpness measures. We compare these measures and show that the low-pass
filter-based measure exhibits the highest correlation with the generalization
abilities of DL models, has high robustness to both data and label noise, and
furthermore can track the double descent behavior for neural networks. We next
derive the optimization algorithm, relying on the low-pass filter (LPF), that
actively searches the flat regions in the DL optimization landscape using
SGD-like procedure. The update of the proposed algorithm, that we call LPF-SGD,
is determined by the gradient of the convolution of the filter kernel with the
loss function and can be efficiently computed using MC sampling. We empirically
show that our algorithm achieves superior generalization performance compared
to the common DL training strategies. On the theoretical front, we prove that
LPF-SGD converges to a better optimal point with smaller generalization error
than SGD.

研究深度学习模型的损失函数梯度下降优化算法，发现低通滤波器滤波后的测量值与 DL 模型的泛化能力最为相关，提出一种基于低通滤波器的优化算法 LPF-SGD，通过梯度与卷积计算实现，与常见的 DL 训练方法相比，该算法在实验中表现出更好的泛化性能，理论上也证明 LPF-SGD 比 SGD 收敛到更好的最优点并具有更小的泛化误差。