As an indispensable component, Batch Normalization (BN) has successfully
improved the training of deep neural networks (DNNs) with mini-batches, by
normalizing the distribution of the internal representation for each hidden
layer. However, the effectiveness of BN would diminish with scenario of
micro-batch (e.g., less than 10 samples in a mini-batch), since the estimated
statistics in a mini-batch are not reliable with insufficient samples. In this
paper, we present a novel normalization method, called Batch Kalman
Normalization (BKN), for improving and accelerating the training of DNNs,
particularly under the context of micro-batches. Specifically, unlike the
existing solutions treating each hidden layer as an isolated system, BKN treats
all the layers in a network as a whole system, and estimates the statistics of
a certain layer by considering the distributions of all its preceding layers,
mimicking the merits of Kalman Filtering. BKN has two appealing properties.
First, it enables more stable training and faster convergence compared to
previous works. Second, training DNNs using BKN performs substantially better
than those using BN and its variants, especially when very small mini-batches
are presented. On the image classification benchmark of ImageNet, using BKN
powered networks we improve upon the best-published model-zoo results: reaching
74.0% top-1 val accuracy for InceptionV2. More importantly, using BKN achieves
the comparable accuracy with extremely smaller batch size, such as 64 times
smaller on CIFAR-10/100 and 8 times smaller on ImageNet.

本文提出了一种称为 Batch Kalman Normalization（BKN）的新型规范化方法，通过考虑其前面所有层的分布来估计某个层的统计信息来改善和加速训练深度神经网络，特别是在微批处理的情况下，实现了更稳定的训练和更快的收敛速度。