Batch Normalization (BN) has become an out-of-box technique to improve deep
network training. However, its effectiveness is limited for micro-batch
training, i.e., each GPU typically has only 1-2 images for training, which is
inevitable for many computer vision tasks, e.g., object detection and semantic
segmentation, constrained by memory consumption. To address this issue, we
propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to
bring two success factors of BN into micro-batch training: 1) the smoothing
effects on the loss landscape and 2) the ability to avoid harmful elimination
singularities along the training trajectory. WS standardizes the weights in
convolutional layers to smooth the loss landscape by reducing the Lipschitz
constants of the loss and the gradients; BCN combines batch and channel
normalizations and leverages estimated statistics of the activations in
convolutional layers to keep networks away from elimination singularities. We
validate WS and BCN on comprehensive computer vision tasks, including image
classification, object detection, instance segmentation, video recognition and
semantic segmentation. All experimental results consistently show that WS and
BCN improve micro-batch training significantly. Moreover, using WS and BCN with
micro-batch training is even able to match or outperform the performances of BN
with large-batch training.

通过使用 Weight Standardization (WS) 和 Batch-Channel Normalization (BCN)，能够将 Batch Normalization（BN）中的两个成功因素引入 micro-batch training 中，从而为计算机视觉任务中不可避免的 micro-batch 训练提供有效的技术支持。通过验证在包括图像分类，目标检测，实例分割，视频识别和语义分割等全面的计算机视觉任务中，WS 和 BCN 均显著改善了 micro-batch 训练，甚至能够与大 batch 训练时的 BN 相媲美或超越。