As models grow larger and more complex, achieving better off-sample
generalization with minimal trial-and-error is critical to the reliability and
economy of machine learning workflows. As a proxy for the well-studied
heuristic of seeking "flat" local minima, gradient regularization is a natural
avenue, and first-order approximations such as Flooding and sharpness-aware
minimization (SAM) have received significant attention, but their performance
depends critically on hyperparameters (flood threshold and neighborhood radius,
respectively) that are non-trivial to specify in advance. In order to develop a
procedure which is more resilient to misspecified hyperparameters, with the
hard-threshold "ascent-descent" switching device used in Flooding as
motivation, we propose a softened, pointwise mechanism called SoftAD that
downweights points on the borderline, limits the effects of outliers, and
retains the ascent-descent effect. We contrast formal stationarity guarantees
with those for Flooding, and empirically demonstrate how SoftAD can realize
classification accuracy competitive with SAM and Flooding while maintaining a
much smaller loss generalization gap and model norm. Our empirical tests range
from simple binary classification on the plane to image classification using
neural networks with millions of parameters; the key trends are observed across
all datasets and models studied, and suggest a potential new approach to
implicit regularization.

通过提出一种更加鲁棒的程序 SoftAD，该程序通过减少边界点和限制异常值的影响，保留升降效果，实现与 SAM 和 Flooding 竞争性分类准确度，同时保持更小的损失泛化差距和模型规范化。