Recently, there has been a surge in interest in developing optimization
algorithms for overparameterized models as achieving generalization is believed
to require algorithms with suitable biases. This interest centers on minimizing
sharpness of the original loss function; the Sharpness-Aware Minimization (SAM)
algorithm has proven effective. However, most literature only considers a few
sharpness measures, such as the maximum eigenvalue or trace of the training
loss Hessian, which may not yield meaningful insights for non-convex
optimization scenarios like neural networks. Additionally, many sharpness
measures are sensitive to parameter invariances in neural networks, magnifying
significantly under rescaling parameters. Motivated by these challenges, we
introduce a new class of sharpness measures in this paper, leading to new
sharpness-aware objective functions. We prove that these measures are
\textit{universally expressive}, allowing any function of the training loss
Hessian matrix to be represented by appropriate hyperparameters. Furthermore,
we show that the proposed objective functions explicitly bias towards
minimizing their corresponding sharpness measures, and how they allow
meaningful applications to models with parameter invariances (such as
scale-invariances). Finally, as instances of our proposed general framework, we
present \textit{Frob-SAM} and \textit{Det-SAM}, which are specifically designed
to minimize the Frobenius norm and the determinant of the Hessian of the
training loss, respectively. We also demonstrate the advantages of our general
framework through extensive experiments.

最近，对于过参数化模型开发优化算法的兴趣不断增加，因为普遍认为实现泛化需要具有适当偏差的算法。本文针对仅考虑少数尖锐度测量值的文献提出了新的尖锐度测量方法，并证明了这些测量方法的普遍表达性，以及它们如何对模型的参数不变性有着显著的影响。此外，本文还提出了 Frob-SAM 和 Det-SAM 两种特定设计的具体优化框架，并通过大量实验证明了该框架的优势。