Batch Normalization (BN) has become an essential technique in contemporary
neural network design, enhancing training stability. Specifically, BN employs
centering and scaling operations to standardize features along the batch
dimension and uses an affine transformation to recover features. Although
standard BN has shown its capability to improve deep neural network training
and convergence, it still exhibits inherent limitations in certain cases. Most
existing techniques that enhance BN consider a single or a few aspects of BN.
In this paper, we first identify problems with BN from a feature perspective
and explore that feature condensation exists in the learning when employing BN,
which negatively affects testing performance. To tackle this problem, we
propose a two-stage unified framework called Unified Batch Normalization (UBN).
In the first stage, we utilize a simple feature condensation threshold to
alleviate the feature condensation, which hinders inappropriate statistic
updates in normalization. In the second stage, we unify various normalization
variants to boost each component of BN. Our experimental results reveal that
UBN significantly enhances performance across different visual backbones and
notably expedites network training convergence, particularly in early training
stages. Notably, our method improved about 3% in top-1 accuracy on ImageNet
classification with large batch sizes, showing the effectiveness of our
approach in real-world scenarios.

本研究论文提出了一种名为统一批标准化（UBN）的两阶段统一框架，用于解决批标准化（BN）中的特征凝聚问题，并统一各种标准化变体以提高性能和网络训练收敛速度。实验证明，UBN 方法能够在不同的视觉背景下显著提升性能，并且在早期训练阶段尤其明显，如在大批量数据上提高了 3% 的 ImageNet 分类的 top-1 准确率，展示了该方法在实际场景中的有效性。