Batch Whitening is a technique that accelerates and stabilizes training by
transforming input features to have a zero mean (Centering) and a unit variance
(Scaling), and by removing linear correlation between channels (Decorrelation).
In commonly used structures, which are empirically optimized with Batch
Normalization, the normalization layer appears between convolution and
activation function. Following Batch Whitening studies have employed the same
structure without further analysis; even Batch Whitening was analyzed on the
premise that the input of a linear layer is whitened. To bridge the gap, we
propose a new Convolutional Unit that is in line with the theory, and our
method generally improves the performance of Batch Whitening. Moreover, we show
the inefficacy of the original Convolutional Unit by investigating rank and
correlation of features. As our method is employable off-the-shelf whitening
modules, we use Iterative Normalization (IterNorm), the state-of-the-art
whitening module, and obtain significantly improved performance on five image
classification datasets: CIFAR-10, CIFAR-100, CUB-200-2011, Stanford Dogs, and
ImageNet. Notably, we verify that our method improves stability and performance
of whitening when using large learning rate, group size, and iteration number.

提出了一种新的卷积单元，可用于批处理美白，并且效果更好，使用迭代归一化并应用于五个图像分类数据集上，显着提高了性能和稳定性。

通过卷积单元优化提升批量白化的泛化能力

Improving Generalization of Batch Whitening by Convolutional Unit  Optimization

Batch Normalization (BN) is extensively employed in various network
architectures by performing standardization within mini-batches.
A full understanding of the process has been a central target in the deep
learning communities.
Unlike existing works, which usually only analyze the standardization
operation, this paper investigates the more general Batch Whitening (BW). Our
work originates from the observation that while various whitening
transformations equivalently improve the conditioning, they show significantly
different behaviors in discriminative scenarios and training Generative
Adversarial Networks (GANs).
We attribute this phenomenon to the stochasticity that BW introduces.
We quantitatively investigate the stochasticity of different whitening
transformations and show that it correlates well with the optimization
behaviors during training.
We also investigate how stochasticity relates to the estimation of population
statistics during inference.
Based on our analysis, we provide a framework for designing and comparing BW
algorithms in different scenarios.
Our proposed BW algorithm improves the residual networks by a significant
margin on ImageNet classification.
Besides, we show that the stochasticity of BW can improve the GAN's
performance with, however, the sacrifice of the training stability.

本文研究批量白化（Batch Whitening），提供了一个可用于不同场景下的批量白化算法设计及比较框架，并通过 ImageNet 数据集分类和生成对抗网络训练的应用实践验证了所提出算法的有效性和稳定性。