Binary neural networks (BNNs) have received increasing attention due to their
superior reductions of computation and memory. Most existing works focus on
either lessening the quantization error by minimizing the gap between the
full-precision weights and their binarization or designing a gradient
approximation to mitigate the gradient mismatch, while leaving the "dead
weights" untouched. This leads to slow convergence when training BNNs. In this
paper, for the first time, we explore the influence of "dead weights" which
refer to a group of weights that are barely updated during the training of
BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead
weights" for updating. We prove that reviving the "dead weights" by ReCU can
result in a smaller quantization error. Besides, we also take into account the
information entropy of the weights, and then mathematically analyze why the
weight standardization can benefit BNNs. We demonstrate the inherent
contradiction between minimizing the quantization error and maximizing the
information entropy, and then propose an adaptive exponential scheduler to
identify the range of the "dead weights". By considering the "dead weights",
our method offers not only faster BNN training, but also state-of-the-art
performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be
available at this https URL

本文通过引入重整夹紧单元（ReCU）同时考虑权重信息熵和死权重，提出了一种针对二值神经网络（BNN）训练的新方法，实现了更快的训练和领先于最近的方法在 CIFAR-10 和 ImageNet 数据集上的性能。