Training an overparameterized neural network can yield minimizers of the same
level of training loss and yet different generalization capabilities. With
evidence that indicates a correlation between sharpness of minima and their
generalization errors, increasing efforts have been made to develop an
optimization method to explicitly find flat minima as more generalizable
solutions. This sharpness-aware minimization (SAM) strategy, however, has not
been studied much yet as to how overparameterization can actually affect its
behavior. In this work, we analyze SAM under varying degrees of
overparameterization and present both empirical and theoretical results that
suggest a critical influence of overparameterization on SAM. Specifically, we
first use standard techniques in optimization to prove that SAM can achieve a
linear convergence rate under overparameterization in a stochastic setting. We
also show that the linearly stable minima found by SAM are indeed flatter and
have more uniformly distributed Hessian moments compared to those of SGD. These
results are corroborated with our experiments that reveal a consistent trend
that the generalization improvement made by SAM continues to increase as the
model becomes more overparameterized. We further present that sparsity can open
up an avenue for effective overparameterization in practice.

训练过参数的神经网络可以得到相同训练损失水平但具有不同泛化能力的极小值。本文分析了过参数化对锐度感知最小化策略（SAM）行为的关键影响，并提供了经验和理论结果，表明过参数化对 SAM 具有重要影响。具体而言，我们证明了在随机设置中 SAM 可以实现线性收敛速度，并且发现 SAM 找到的具有线性稳定性的极小值相比 SGD 更加平坦且具有更均匀分布的 Hessian 矩。我们的实验结果进一步表明，随着模型过参数化程度的增加，SAM 的泛化性能持续改善。我们还展示了稀疏性在实践中为有效的过参数化提供了途径。

过度参数化对锐度感知最小化的影响：实证与理论分析

The Effects of Overparameterization on Sharpness-aware Minimization: An  Empirical and Theoretical Analysis

Overparameterized neural networks (NNs) are observed to generalize well even
when trained to perfectly fit noisy data. This phenomenon motivated a large
body of work on "benign overfitting", where interpolating predictors achieve
near-optimal performance. Recently, it was conjectured and empirically observed
that the behavior of NNs is often better described as "tempered overfitting",
where the performance is non-optimal yet also non-trivial, and degrades as a
function of the noise level. However, a theoretical justification of this claim
for non-linear NNs has been lacking so far. In this work, we provide several
results that aim at bridging these complementing views. We study a simple
classification setting with 2-layer ReLU NNs, and prove that under various
assumptions, the type of overfitting transitions from tempered in the extreme
case of one-dimensional data, to benign in high dimensions. Thus, we show that
the input dimension has a crucial role on the type of overfitting in this
setting, which we also validate empirically for intermediate dimensions.
Overall, our results shed light on the intricate connections between the
dimension, sample size, architecture and training algorithm on the one hand,
and the type of resulting overfitting on the other hand.

本文研究过参数神经网络的过拟合现象，并证明了在不同的维度下会出现不同种类的过拟合现象，包括良性过拟合和温和过拟合，并解释了这些现象与样本大小、网络结构等因素的相关性。