The best performing Binary Neural Networks (BNNs) are usually attained using Adam optimization and its multi-step training variants. However, to the best of our knowledge, few studies explore the fundamental reasons why Adam is superior to other optimizers like SGD for BNN optimization or provide analytical explanations that support specific training strategies. To address this, in this paper we first investigate the trajectories of gradients and weights in BNNs during the training process. We show the regularization effect of second-order momentum in Adam is crucial to revitalize the weights that are dead due to the activation saturation in BNNs. We find that Adam, through its adaptive learning rate strategy, is better equipped to handle the rugged loss surface of BNNs and reaches a better optimum with higher generalization ability. Furthermore, we inspect the intriguing role of the real-valued weights in binary networks, and reveal the effect of weight decay on the stability and sluggishness of BNN optimization. Through extensive experiments and analysis, we derive a simple training scheme, building on existing Adam-based optimization, which achieves 70.5% top-1 accuracy on the ImageNet dataset using the same architecture as the state-of-the-art ReActNet while achieving 1.1% higher accuracy. Code and models are available at https://github.com/liuzechun/AdamBNN.

研究二进制神经网络使用Adam优化相对于SGD优化的优势，发现Adam通过其自适应学习率策略更好地处理BNN的崎岖损失表面，并得到更好的优化结果。通过分析发现，Adam的二阶动量正则化效应对于使BNN中由于激活饱和而死亡的权重重振是至关重要的，并探究了实值权重在二进制网络中的有趣作用以及权重衰减对BNN优化的影响。最终，我们提出了一个基于Adam优化的简单训练方案，使用相同的架构比最先进的ReActNet实现了1.1％更高的top-1准确度（70.5％）。

Adam与训练策略如何帮助BNNs优化？