In neural network binarization, BinaryConnect (BC) and its variants are
considered the standard. These methods apply the sign function in their forward
pass and their respective gradients are backpropagated to update the weights.
However, the derivative of the sign function is zero whenever defined, which
consequently freezes training. Therefore, implementations of BC (e.g., BNN)
usually replace the derivative of sign in the backward computation with
identity or other approximate gradient alternatives. Although such practice
works well empirically, it is largely a heuristic or ''training trick.'' We aim
at shedding some light on these training tricks from the optimization
perspective. Building from existing theory on ProxConnect (PC, a generalization
of BC), we (1) equip PC with different forward-backward quantizers and obtain
ProxConnect++ (PC++) that includes existing binarization techniques as special
cases; (2) derive a principled way to synthesize forward-backward quantizers
with automatic theoretical guarantees; (3) illustrate our theory by proposing
an enhanced binarization algorithm BNN++; (4) conduct image classification
experiments on CNNs and vision transformers, and empirically verify that BNN++
generally achieves competitive results on binarizing these models.

通过对于神经网络二值化的训练技巧进行优化，建立了 ProxConnect++ (PC++) 作为 BinaryConnect (BC) 的扩展，提出了一种增强的二值化算法 BNN++，并在卷积神经网络和视觉转换模型的图像分类实验中进行了验证。

使用前向和后向近似量化理解神经网络二值化

Understanding Neural Network Binarization with Forward and Backward  Proximal Quantizers

Binarization of neural networks is a dominant paradigm in neural networks
compression. The pioneering work BinaryConnect uses Straight Through Estimator
(STE) to mimic the gradients of the sign function, but it also causes the
crucial inconsistency problem. Most of the previous methods design different
estimators instead of STE to mitigate it. However, they ignore the fact that
when reducing the estimating error, the gradient stability will decrease
concomitantly. These highly divergent gradients will harm the model training
and increase the risk of gradient vanishing and gradient exploding. To fully
take the gradient stability into consideration, we present a new perspective to
the BNNs training, regarding it as the equilibrium between the estimating error
and the gradient stability. In this view, we firstly design two indicators to
quantitatively demonstrate the equilibrium phenomenon. In addition, in order to
balance the estimating error and the gradient stability well, we revise the
original straight through estimator and propose a power function based
estimator, Rectified Straight Through Estimator (ReSTE for short). Comparing to
other estimators, ReSTE is rational and capable of flexibly balancing the
estimating error with the gradient stability. Extensive experiments on CIFAR-10
and ImageNet datasets show that ReSTE has excellent performance and surpasses
the state-of-the-art methods without any auxiliary modules or losses.

通过平衡估计误差和梯度稳定性，我们提出了修正的直通估计器（ReSTE），用于神经网络压缩，其比其他估计器更合理且能灵活地平衡估计误差和梯度稳定性。实验结果表明，ReSTE 在 CIFAR-10 和 ImageNet 数据集上表现出色，超越了其他现有方法，而不需要任何辅助模块或损失函数。

估计器遇上平衡视角：用于训练二值神经网络的修正直通估计器

Estimator Meets Equilibrium Perspective: A Rectified Straight Through  Estimator for Binary Neural Networks Training

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide
range of tasks, with the best results obtained with large training sets and
large models. In the past, GPUs enabled these breakthroughs because of their
greater computational speed. In the future, faster computation at both training
and test time is likely to be crucial for further progress and for consumer
applications on low-power devices. As a result, there is much interest in
research and development of dedicated hardware for Deep Learning (DL). Binary
weights, i.e., weights which are constrained to only two possible values (e.g.
-1 or 1), would bring great benefits to specialized DL hardware by replacing
many multiply-accumulate operations by simple accumulations, as multipliers are
the most space and power-hungry components of the digital implementation of
neural networks. We introduce BinaryConnect, a method which consists in
training a DNN with binary weights during the forward and backward
propagations, while retaining precision of the stored weights in which
gradients are accumulated. Like other dropout schemes, we show that
BinaryConnect acts as regularizer and we obtain near state-of-the-art results
with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

该研究介绍了一种名为 BinaryConnect 的方法，该方法通过对神经网络进行二进制权重的训练和保留梯度累积精度的方式，取代传统神经网络中高昂代价的乘法运算，提高特定硬件上的深度学习性能，并取得了接近最优的结果。