Batch Normalization (BN) is one of the most widely used techniques in Deep
Learning field. But its performance can awfully degrade with insufficient batch
size. This weakness limits the usage of BN on many computer vision tasks like
detection or segmentation, where batch size is usually small due to the
constraint of memory consumption. Therefore many modified normalization
techniques have been proposed, which either fail to restore the performance of
BN completely, or have to introduce additional nonlinear operations in
inference procedure and increase huge consumption. In this paper, we reveal
that there are two extra batch statistics involved in backward propagation of
BN, on which has never been well discussed before. The extra batch statistics
associated with gradients also can severely affect the training of deep neural
network. Based on our analysis, we propose a novel normalization method, named
Moving Average Batch Normalization (MABN). MABN can completely restore the
performance of vanilla BN in small batch cases, without introducing any
additional nonlinear operations in inference procedure. We prove the benefits
of MABN by both theoretical analysis and experiments. Our experiments
demonstrate the effectiveness of MABN in multiple computer vision tasks
including ImageNet and COCO. The code has been released in
this https URL

本文提出一种新的归一化方法，即移动平均批量归一化（MABN），可以在小批量情况下完全恢复基本 BN 的性能，并且在推理过程中不需要引入任何额外的非线性操作，此方法通过理论分析和实验演示了其有效性。