Normalization layers (e.g., Batch Normalization, Layer Normalization) were
introduced to help with optimization difficulties in very deep nets, but they
clearly also help generalization, even in not-so-deep nets. Motivated by the
long-held belief that flatter minima lead to better generalization, this paper
gives mathematical analysis and supporting experiments suggesting that
normalization (together with accompanying weight-decay) encourages GD to reduce
the sharpness of loss surface. Here "sharpness" is carefully defined given that
the loss is scale-invariant, a known consequence of normalization.
Specifically, for a fairly broad class of neural nets with normalization, our
theory explains how GD with a finite learning rate enters the so-called Edge of
Stability (EoS) regime, and characterizes the trajectory of GD in this regime
via a continuous sharpness-reduction flow.

通过数学分析和实验证明，在深度网络中引入标准化层（例如批量标准化，层标准化）有利于优化并促进泛化，同时对于包含标准化的一类神经网络，伴随权值衰减的标准化可以鼓励梯度下降到达稳定边缘，并且对于这种情况，可以确定梯度下降的流动轨迹。

理解归一化层的泛化增益：锐化减少

Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction

Batch Normalization (BN) is a commonly used technique to accelerate and
stabilize training of deep neural networks. Despite its empirical success, a
full theoretical understanding of BN is yet to be developed. In this work, we
analyze BN through the lens of convex optimization. We introduce an analytic
framework based on convex duality to obtain exact convex representations of
weight-decay regularized ReLU networks with BN, which can be trained in
polynomial-time. Our analyses also show that optimal layer weights can be
obtained as simple closed-form formulas in the high-dimensional and/or
overparameterized regimes. Furthermore, we find that Gradient Descent provides
an algorithmic bias effect on the standard non-convex BN network, and we design
an approach to explicitly encode this implicit regularization into the convex
objective. Experiments with CIFAR image classification highlight the
effectiveness of this explicit regularization for mimicking and substantially
improving the performance of standard BN networks.

本文通过凸优化的视角分析 Batch Normalization，提出了一个基于凸对偶的解析框架，可以精确地描述用 Batch Normalization 训练的带有权重衰减的 ReLU 网络，并证明在高维和过参数化情况下，理论上可以获得一些简单的解析的最优层权重和可训练的凸约束优化问题，并发现梯度下降给标准的非凸 BN 网络提供了算法偏差效应，通过我们的方法可以将这种隐式正则化显性编码到凸目标中，实验结果表明该方法可以模拟和显著提高标准 BN 网络的性能。

ReLU 网络中批归一化的解析：等价的凸优化模型与隐式正则化

Demystifying Batch Normalization in ReLU Networks: Equivalent Convex  Optimization Models and Implicit Regularization

Over the past few years, Batch-Normalization has been commonly used in deep
networks, allowing faster training and high performance for a wide variety of
applications. However, the reasons behind its merits remained unanswered, with
several shortcomings that hindered its use for certain tasks. In this work, we
present a novel view on the purpose and function of normalization methods and
weight-decay, as tools to decouple weights' norm from the underlying optimized
objective. This property highlights the connection between practices such as
normalization, weight decay and learning-rate adjustments. We suggest several
alternatives to the widely used $L^2$ batch-norm, using normalization in $L^1$
and $L^\infty$ spaces that can substantially improve numerical stability in
low-precision implementations as well as provide computational and memory
benefits. We demonstrate that such methods enable the first batch-norm
alternative to work for half-precision implementations. Finally, we suggest a
modification to weight-normalization, which improves its performance on
large-scale tasks.

本研究提出了一种新的对归一化方法和权值衰减的目的和功能的视角，并建议了几种常用 L2 批归一化的替代方案，包括 L1 和 L∞范数中的归一化，以提高计算和内存效率，并且在低精度实现中大幅提高了数值稳定性，并且还提出了一种改进重要性重归一化的方法，可以提高其在大规模任务上的性能。