In this paper, we study 1-bit convolutional neural networks (CNNs), of which
both the weights and activations are binary. While efficient, the lacking of
representational capability and the training difficulty impede 1-bit CNNs from
performing as well as real-valued networks. We propose Bi-Real net with a novel
training algorithm to tackle these two challenges. To enhance the
representational capability, we propagate the real-valued activations generated
by each 1-bit convolution via a parameter-free shortcut. To address the
training difficulty, we propose a training algorithm using a tighter
approximation to the derivative of the sign function, a magnitude-aware
gradient for weight updating, a better initialization method, and a two-step
scheme for training a deep network. Experiments on ImageNet show that an
18-layer Bi-Real net with the proposed training algorithm achieves 56.4% top-1
classification accuracy, which is 10% higher than the state-of-the-arts (e.g.,
XNOR-Net) with greater memory saving and lower computational cost. Bi-Real net
is also the first to scale up 1-bit CNNs to an ultra-deep network with 152
layers, and achieves 64.5% top-1 accuracy on ImageNet. A 50-layer Bi-Real net
shows comparable performance to a real-valued network on the depth estimation
task with only a 0.3% accuracy gap.

本文研究了一比特卷积神经网络，通过使用 Bi-Real net 和新颖的训练算法来提高其表现能力和训练困难性，并在 ImageNet 上获得了 56.4％的 top-1 分类准确性，比现有技术提高了 10％以上。

Bi-Real Net: 将深度网络二值化以实现真实网络性能

Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance

In this work, we study the 1-bit convolutional neural networks (CNNs), of
which both the weights and activations are binary. While being efficient, the
classification accuracy of the current 1-bit CNNs is much worse compared to
their counterpart real-valued CNN models on the large-scale dataset, like
ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN
models, we propose a novel model, dubbed Bi-Real net, which connects the real
activations (after the 1-bit convolution and/or BatchNorm layer, before the
sign function) to activations of the consecutive block, through an identity
shortcut. Consequently, compared to the standard 1-bit CNN, the
representational capability of the Bi-Real net is significantly enhanced and
the additional cost on computation is negligible. Moreover, we develop a
specific training algorithm including three technical novelties for 1- bit
CNNs. Firstly, we derive a tight approximation to the derivative of the
non-differentiable sign function with respect to activation. Secondly, we
propose a magnitude-aware gradient with respect to the weight for updating the
weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip
function, rather than the ReLU function, to better initialize the Bi-Real net.
Experiments on ImageNet show that the Bi-Real net with the proposed training
algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers,
respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net
achieves up to 10% higher top-1 accuracy with more memory saving and lower
computational cost. Keywords: binary neural network, 1-bit CNNs,
1-layer-per-block

提出了 Bi-Real net 模型，它采用了一种新的训练算法来提高 1-bit CNNs 对大规模数据集 ImageNet 分类的性能，并在 18 层和 34 层下达到了 56.4% 和 62.2% 的 top-1 准确率。