Binarized Neural Networks (BNNs) can significantly reduce the inference
latency and energy consumption in resource-constrained devices due to their
pure-logical computation and fewer memory accesses. However, training BNNs is
difficult since the activation flow encounters degeneration, saturation, and
gradient mismatch problems. Prior work alleviates these issues by increasing
activation bits and adding floating-point scaling factors, thereby sacrificing
BNN's energy efficiency. In this paper, we propose to use distribution loss to
explicitly regularize the activation flow, and develop a framework to
systematically formulate the loss. Our experiments show that the distribution
loss can consistently improve the accuracy of BNNs without losing their energy
benefits. Moreover, equipped with the proposed regularization, BNN training is
shown to be robust to the selection of hyper-parameters including optimizer and
learning rate.

本文提出使用分布损失来规范激活流，成功改善了二值神经网络训练中遇到的退化、饱和和梯度失配等问题，并在不影响其能量效率的同时，提高了其准确性。此外，该规范化方法还能使训练过程更加稳健。