Toward achieving robust and defensive neural networks, the robustness against
the weight parameters perturbations, i.e., sharpness, attracts attention in
recent years (Sun et al., 2020). However, sharpness is known to remain a
critical issue, "scale-sensitivity." In this paper, we propose a novel
sharpness measure, Minimum Sharpness. It is known that NNs have a specific
scale transformation that constitutes equivalent classes where functional
properties are completely identical, and at the same time, their sharpness
could change unlimitedly. We define our sharpness through a minimization
problem over the equivalent NNs being invariant to the scale transformation. We
also develop an efficient and exact technique to make the sharpness tractable,
which reduces the heavy computational costs involved with Hessian. In the
experiment, we observed that our sharpness has a valid correlation with the
generalization of NNs and runs with less computational cost than existing
sharpness measures.

本文提出了一种基于最小锐度的新的锐度测量方法，旨在解决规模敏感性问题，定义在等效的神经网络上，与常用锐度测量方法相比具有更小的计算代价和更好的一般化性能。

最小锐度：神经网络的尺度不变参数鲁棒性

Minimum sharpness: Scale-invariant parameter-robustness of neural  networks

It is arguably believed that flatter minima can generalize better. However,
it has been pointed out that the usual definitions of sharpness, which consider
either the maxima or the integral of loss over a $\delta$ ball of parameters
around minima, cannot give consistent measurement for scale invariant neural
networks, e.g., networks with batch normalization layer. In this paper, we
first propose a measure of sharpness, BN-Sharpness, which gives consistent
value for equivalent networks under BN. It achieves the property of scale
invariance by connecting the integral diameter with the scale of parameter.
Then we present a computation-efficient way to calculate the BN-sharpness
approximately i.e., one dimensional integral along the "sharpest" direction.
Furthermore, we use the BN-sharpness to regularize the training and design an
algorithm to minimize the new regularized objective. Our algorithm achieves
considerably better performance than vanilla SGD over various experiment
settings.

本文提出了一种与 BN 层等价的神经网络下具有一致性的、刻画了梯度的度量标准和对应的优化算法，该算法利用该标准对训练进行正则化，获得了与原始 SGD 相比较显著的性能提升。