Binary neural networks (BNNs) have received ever-increasing popularity for
their great capability of reducing storage burden as well as quickening
inference time. However, there is a severe performance drop compared with
real-valued networks, due to its intrinsic frequent weight oscillation during
training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN)
to mitigate the frequent oscillation for better BNNs' training. We identify
that the weight oscillation mainly stems from the non-parametric scaling
factor. To address this issue, we propose to parameterize the scaling factor
and introduce a weighted reconstruction loss to build an adaptive training
objective. For the first time, we show that the weight oscillation is
controlled by the balanced parameter attached to the reconstruction loss, which
provides a theoretical foundation to parameterize it in back propagation. Based
on this, we learn our ReBNN by calculating the balanced parameter based on its
maximum magnitude, which can effectively mitigate the weight oscillation with a
resilient training process. Extensive experiments are conducted upon various
network models, such as ResNet and Faster-RCNN for computer vision, as well as
BERT for natural language processing. The results demonstrate the overwhelming
performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9%
Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing
existing state-of-the-arts by a significant margin. Our code is open-sourced at
this https URL

本文提出了一种 Resilient Binary Neural Network（ReBNN）来减轻二进制神经网络的重量震荡问题，并通过引入加权重建损失来构建自适应训练目标，该方法在计算机视觉等多种网络模型上具有优异的表现。