Batch Whitening is a technique that accelerates and stabilizes training by
transforming input features to have a zero mean (Centering) and a unit variance
(Scaling), and by removing linear correlation between channels (Decorrelation).
In commonly used structures, which are empirically optimized with Batch
Normalization, the normalization layer appears between convolution and
activation function. Following Batch Whitening studies have employed the same
structure without further analysis; even Batch Whitening was analyzed on the
premise that the input of a linear layer is whitened. To bridge the gap, we
propose a new Convolutional Unit that is in line with the theory, and our
method generally improves the performance of Batch Whitening. Moreover, we show
the inefficacy of the original Convolutional Unit by investigating rank and
correlation of features. As our method is employable off-the-shelf whitening
modules, we use Iterative Normalization (IterNorm), the state-of-the-art
whitening module, and obtain significantly improved performance on five image
classification datasets: CIFAR-10, CIFAR-100, CUB-200-2011, Stanford Dogs, and
ImageNet. Notably, we verify that our method improves stability and performance
of whitening when using large learning rate, group size, and iteration number.

提出了一种新的卷积单元，可用于批处理美白，并且效果更好，使用迭代归一化并应用于五个图像分类数据集上，显着提高了性能和稳定性。

通过卷积单元优化提升批量白化的泛化能力

Improving Generalization of Batch Whitening by Convolutional Unit  Optimization

Batch Normalization (BN) is a popular technique for training Deep Neural
Networks (DNNs). BN uses scaling and shifting to normalize activations of
mini-batches to accelerate convergence and improve generalization. The recently
proposed Iterative Normalization (IterNorm) method improves these properties by
whitening the activations iteratively using Newton's method. However, since
Newton's method initializes the whitening matrix independently at each training
step, no information is shared between consecutive steps. In this work, instead
of exact computation of whitening matrix at each time step, we estimate it
gradually during training in an online fashion, using our proposed Stochastic
Whitening Batch Normalization (SWBN) algorithm. We show that while SWBN
improves the convergence rate and generalization of DNNs, its computational
overhead is less than that of IterNorm. Due to the high efficiency of the
proposed method, it can be easily employed in most DNN architectures with a
large number of layers. We provide comprehensive experiments and comparisons
between BN, IterNorm, and SWBN layers to demonstrate the effectiveness of the
proposed technique in conventional (many-shot) image classification and
few-shot classification tasks.

本研究提出了一种名为 SWBN 的针对深度神经网络的新算法，该算法使用随机白化技术渐进估计白化矩阵，相较于 IterNorm 算法实现了更快的收敛和更好的泛化效果。在常规（大批量）图像分类和少样本分类任务上，经过综合实验和对比，显示出了 SWBN 方法的有效性。

随机白化批量归一化

Stochastic Whitening Batch Normalization

Cross-lingual word embeddings (CLWE) underlie many multilingual natural
language processing systems, often through orthogonal transformations of
pre-trained monolingual embeddings. However, orthogonal mapping only works on
language pairs whose embeddings are naturally isomorphic. For non-isomorphic
pairs, our method (Iterative Normalization) transforms monolingual embeddings
to make orthogonal alignment easier by simultaneously enforcing that (1)
individual word vectors are unit length, and (2) each language's average vector
is zero. Iterative Normalization consistently improves word translation
accuracy of three CLWE methods, with the largest improvement observed on
English-Japanese (from 2% to 44% test accuracy).

通过迭代正则化方法，使得非同构语言 pairs 的单语词嵌入向量单位化和对齐，从而提高跨语言词嵌入的翻译准确度，特别是在英文 - 日文 pairs 中，测试准确性从 2% 提高到 44%。

女孩是猫娘还是少女？迭代规范化法实现非同构嵌入的跨语言对齐

Are Girls Neko or Shōjo? Cross-Lingual Alignment of Non-Isomorphic  Embeddings with Iterative Normalization

Batch Normalization (BN) is ubiquitously employed for accelerating neural
network training and improving the generalization capability by performing
standardization within mini-batches. Decorrelated Batch Normalization (DBN)
further boosts the above effectiveness by whitening. However, DBN relies
heavily on either a large batch size, or eigen-decomposition that suffers from
poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which
employs Newton's iterations for much more efficient whitening, while
simultaneously avoiding the eigen-decomposition. Furthermore, we develop a
comprehensive study to show IterNorm has better trade-off between optimization
and generalization, with theoretical and experimental support. To this end, we
exclusively introduce Stochastic Normalization Disturbance (SND), which
measures the inherent stochastic uncertainty of samples when applied to
normalization operations. With the support of SND, we provide natural
explanations to several phenomena from the perspective of optimization, e.g.,
why group-wise whitening of DBN generally outperforms full-whitening and why
the accuracy of BN degenerates with reduced batch sizes. We demonstrate the
consistently improved performance of IterNorm with extensive experiments on
CIFAR-10 and ImageNet over BN and DBN.

本文提出了一种新的神经网络训练方法：Iterative Normalization，通过牛顿迭代实现高效且无矩阵分解的白化，同时引入 Stochastic Normalization Disturbance，增加实验的稳定性，实验表明 IterNorm 具有更好的优化与泛化折衷。