The optimization of Binary Neural Networks (BNNs) relies on approximating the
real-valued weights with their binarized representations. Current techniques
for weight-updating use the same approaches as traditional Neural Networks
(NNs) with the extra requirement of using an approximation to the derivative of
the sign function - as it is the Dirac-Delta function - for back-propagation;
thus, efforts are focused adapting full-precision techniques to work on BNNs.
In the literature, only one previous effort has tackled the problem of directly
training the BNNs with bit-flips by using the first raw moment estimate of the
gradients and comparing it against a threshold for deciding when to flip a
weight (Bop). In this paper, we take an approach parallel to Adam which also
uses the second raw moment estimate to normalize the first raw moment before
doing the comparison with the threshold, we call this method Bop2ndOrder. We
present two versions of the proposed optimizer: a biased one and a
bias-corrected one, each with its own applications. Also, we present a complete
ablation study of the hyperparameters space, as well as the effect of using
schedulers on each of them. For these studies, we tested the optimizer in
CIFAR10 using the BinaryNet architecture. Also, we tested it in ImageNet 2012
with the XnorNet and BiRealNet architectures for accuracy. In both datasets our
approach proved to converge faster, was robust to changes of the
hyperparameters, and achieved better accuracy values.

本文提出了一种用于训练 BNN 的优化器方法 Bop2ndOrder，使用双曲正切函数对二值化的权重进行优化，在超参数空间的完整消融研究及对比实验中，该方法在 CIFAR10 和 ImageNet 数据集中都能够更快地收敛并获得更好的准确性。