This study introduces a new normalization layer termed Batch Layer
Normalization (BLN) to reduce the problem of internal covariate shift in deep
neural network layers. As a combined version of batch and layer normalization,
BLN adaptively puts appropriate weight on mini-batch and feature normalization
based on the inverse size of mini-batches to normalize the input to a layer
during the learning process. It also performs the exact computation with a
minor change at inference times, using either mini-batch statistics or
population statistics. The decision process to either use statistics of
mini-batch or population gives BLN the ability to play a comprehensive role in
the hyper-parameter optimization process of models. The key advantage of BLN is
the support of the theoretical analysis of being independent of the input data,
and its statistical configuration heavily depends on the task performed, the
amount of training data, and the size of batches. Test results indicate the
application potential of BLN and its faster convergence than batch
normalization and layer normalization in both Convolutional and Recurrent
Neural Networks. The code of the experiments is publicly available online
(this https URL).

介绍了一种新的归一化层 Batch Layer Normalization（BLN），可以在深度神经网络中减少内部协变量偏移问题，通过适应性的权衡 mini-batch 和特征标准化，并且具有比批归一化和层归一化更快的收敛速度。