Binary neural networks (BNNs) have received increasing attention due to their
superior reductions of computation and memory. Most existing works focus on
either lessening the quantization error by minimizing the gap between the
full-precision weights and their binarization or designing a gradient
approximation to mitigate the gradient mismatch, while leaving the "dead
weights" untouched. This leads to slow convergence when training BNNs. In this
paper, for the first time, we explore the influence of "dead weights" which
refer to a group of weights that are barely updated during the training of
BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead
weights" for updating. We prove that reviving the "dead weights" by ReCU can
result in a smaller quantization error. Besides, we also take into account the
information entropy of the weights, and then mathematically analyze why the
weight standardization can benefit BNNs. We demonstrate the inherent
contradiction between minimizing the quantization error and maximizing the
information entropy, and then propose an adaptive exponential scheduler to
identify the range of the "dead weights". By considering the "dead weights",
our method offers not only faster BNN training, but also state-of-the-art
performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be
available at this https URL

本文通过引入重整夹紧单元（ReCU）同时考虑权重信息熵和死权重，提出了一种针对二值神经网络（BNN）训练的新方法，实现了更快的训练和领先于最近的方法在 CIFAR-10 和 ImageNet 数据集上的性能。

ReCU：二进制神经网络中死亡权重的复活

ReCU: Reviving the Dead Weights in Binary Neural Networks

Batch Normalization is a key component in almost all state-of-the-art image
classifiers, but it also introduces practical challenges: it breaks the
independence between training examples within a batch, can incur compute and
memory overhead, and often results in unexpected bugs. Building on recent
theoretical analyses of deep ResNets at initialization, we propose a simple set
of analysis tools to characterize signal propagation on the forward pass, and
leverage these tools to design highly performant ResNets without activation
normalization layers. Crucial to our success is an adapted version of the
recently proposed Weight Standardization. Our analysis tools show how this
technique preserves the signal in networks with ReLU or Swish activation
functions by ensuring that the per-channel activation means do not grow with
depth. Across a range of FLOP budgets, our networks attain performance
competitive with the state-of-the-art EfficientNets on ImageNet.

该研究介绍了一种能够设计高效 ResNets 网络的分析工具，通过采用一种改进后的 Weight Standardization 技术，该分析工具的作用在于确保具有 ReLU 或 Swish 激活函数的信号在网络的深度方向上不发生偏移以达到在 FLOP 预算范围内与当今最先进的 EfficientNets 在 ImageNet 上的性能竞争可比性。

特征化信号传播以缩小未标准化 ResNets 性能差距

Characterizing signal propagation to close the performance gap in  unnormalized ResNets

Batch Normalization (BN) has become an out-of-box technique to improve deep
network training. However, its effectiveness is limited for micro-batch
training, i.e., each GPU typically has only 1-2 images for training, which is
inevitable for many computer vision tasks, e.g., object detection and semantic
segmentation, constrained by memory consumption. To address this issue, we
propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to
bring two success factors of BN into micro-batch training: 1) the smoothing
effects on the loss landscape and 2) the ability to avoid harmful elimination
singularities along the training trajectory. WS standardizes the weights in
convolutional layers to smooth the loss landscape by reducing the Lipschitz
constants of the loss and the gradients; BCN combines batch and channel
normalizations and leverages estimated statistics of the activations in
convolutional layers to keep networks away from elimination singularities. We
validate WS and BCN on comprehensive computer vision tasks, including image
classification, object detection, instance segmentation, video recognition and
semantic segmentation. All experimental results consistently show that WS and
BCN improve micro-batch training significantly. Moreover, using WS and BCN with
micro-batch training is even able to match or outperform the performances of BN
with large-batch training.

通过使用 Weight Standardization (WS) 和 Batch-Channel Normalization (BCN)，能够将 Batch Normalization（BN）中的两个成功因素引入 micro-batch training 中，从而为计算机视觉任务中不可避免的 micro-batch 训练提供有效的技术支持。通过验证在包括图像分类，目标检测，实例分割，视频识别和语义分割等全面的计算机视觉任务中，WS 和 BCN 均显著改善了 micro-batch 训练，甚至能够与大 batch 训练时的 BN 相媲美或超越。