In many recent works, there is an increased focus on designing algorithms
that seek flatter optima for neural network loss optimization as there is
empirical evidence that it leads to better generalization performance in many
datasets. In this work, we dissect these performance gains through the lens of
data memorization in overparameterized models. We define a new metric that
helps us identify which data points specifically do algorithms seeking flatter
optima do better when compared to vanilla SGD. We find that the generalization
gains achieved by Sharpness Aware Minimization (SAM) are particularly
pronounced for atypical data points, which necessitate memorization. This
insight helps us unearth higher privacy risks associated with SAM, which we
verify through exhaustive empirical evaluations. Finally, we propose mitigation
strategies to achieve a more desirable accuracy vs privacy tradeoff.

我们通过数据存储在过参数化模型中的方式来研究寻求更平坦的神经网络损失优化算法如何导致更好的泛化性能，我们提出了新的指标来帮助我们确定哪些数据点在与普通 SGD 相比寻求更平坦最优解的算法中表现更好。我们发现了 Sharpness Aware Minimization (SAM) 所实现的泛化性能提升特别明显的非典型数据点，这需要数据的存储。这一观点帮助我们发现了与 SAM 相关的更高的隐私风险，我们通过详尽的实证评估进行了验证。最后，我们提出了缓解策略以实现更理想的准确性与隐私权衡。