Normalization techniques have become a basic component in modern
convolutional neural networks (ConvNets). In particular, many recent works
demonstrate that promoting the orthogonality of the weights helps train deep
models and improve robustness. For ConvNets, most existing methods are based on
penalizing or normalizing weight matrices derived from concatenating or
flattening the convolutional kernels. These methods often destroy or ignore the
benign convolutional structure of the kernels; therefore, they are often
expensive or impractical for deep ConvNets. In contrast, we introduce a simple
and efficient "Convolutional Normalization" (ConvNorm) method that can fully
exploit the convolutional structure in the Fourier domain and serve as a simple
plug-and-play module to be conveniently incorporated into any ConvNets. Our
method is inspired by recent work on preconditioning methods for convolutional
sparse coding and can effectively promote each layer's channel-wise isometry.
Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm
of the weight matrices and hence improve the Lipschitzness of the network,
leading to easier training and improved robustness for deep ConvNets. Applied
to classification under noise corruptions and generative adversarial network
(GAN), we show that the ConvNorm improves the robustness of common ConvNets
such as ResNet and the performance of GAN. We verify our findings via numerical
experiments on CIFAR and ImageNet.

介绍了一种基于卷积结构的归一化方法 (Convolutional Normalization)，它可以方便地作为模块嵌入到任何卷积神经网络 (ConvNets) 中，通过提高权重的正交性和每层的通道等距性，减小权重矩阵的层范数，提高网络的 Lipschitzness 并且提高 GAN 性能和普适性。