We systematically explored a spectrum of normalization algorithms related to
Batch Normalization (BN) and propose a generalized formulation that
simultaneously solves two major limitations of BN: (1) online learning and (2)
recurrent learning. Our proposal is simpler and more biologically-plausible.
Unlike previous approaches, our technique can be applied out of the box to all
learning scenarios (e.g., online learning, batch learning, fully-connected,
convolutional, feedforward, recurrent and mixed --- recurrent and
convolutional) and compare favorably with existing approaches. We also propose
Lp Normalization for normalizing by different orders of statistical moments. In
particular, L1 normalization is well-performing, simple to implement, fast to
compute, more biologically-plausible and thus ideal for GPU or hardware
implementations.

提出了一个更简单、更符合生物学规律的通用归一化算法，可同时解决批归一化的两个主要限制：在线学习和循环学习，并提出了用不同统计矩阶数来进行归一化的 Lp 归一化方法，特别是 L1 归一化，其性能表现良好且计算速度快，更加符合生物学规律，因此非常适合 GPU 或硬件实现。