We perform an extensive empirical study of the statistical properties of
Batch Norm and other common normalizers. This includes an examination of the
correlation between representations of minibatches, gradient norms, and Hessian
spectra both at initialization and over the course of training. Through this
analysis, we identify several statistical properties which appear linked to
Batch Norm's superior performance. We propose two simple normalizers,
PreLayerNorm and RegNorm, which better match these desirable properties without
involving operations along the batch dimension. We show that PreLayerNorm and
RegNorm achieve much of the performance of Batch Norm without requiring batch
dependence, that they reliably outperform LayerNorm, and that they can be
applied in situations where Batch Norm is ineffective.

本论文通过对 “Batch Norm” 等常见标准化方法的统计特性进行广泛的实证研究，提出可以更好地匹配这些理想特性的两种新的标准化方法，PreLayerNorm 和 RegNorm 。研究表明，它们可以实现 Batch Norm 的大部分性能，而无需批次依赖关系，并且它们可在 Batch Norm 无效的情况下应用。