Batch Normalization (BN) has been a standard component in designing deep
neural networks (DNNs). Although the standard BN can significantly accelerate
the training of DNNs and improve the generalization performance, it has several
underlying limitations which may hamper the performance in both training and
inference. In the training stage, BN relies on estimating the mean and variance
of data using a single minibatch. Consequently, BN can be unstable when the
batch size is very small or the data is poorly sampled. In the inference stage,
BN often uses the so called moving mean and moving variance instead of batch
statistics, i.e., the training and inference rules in BN are not consistent.
Regarding these issues, we propose a memorized batch normalization (MBN), which
considers multiple recent batches to obtain more accurate and robust
statistics. Note that after the SGD update for each batch, the model parameters
will change, and the features will change accordingly, leading to the
Distribution Shift before and after the update for the considered batch. To
alleviate this issue, we present a simple Double-Forward scheme in MBN which
can further improve the performance. Compared to related methods, the proposed
MBN exhibits consistent behaviors in both training and inference. Empirical
results show that the MBN based models trained with the Double-Forward scheme
greatly reduce the sensitivity of data and significantly improve the
generalization performance.

本文提出了一种基于多个最近批次来获取更准确、更稳健统计的 Memorized Batch Normalization (MBN)，并使用 Double-Forward scheme 来缓解分布漂移问题，相较于现有的方法，在训练和推理中表现更加稳定，并显著提高了模型的泛化性能。