Binary Neural Networks (BNNs) are compact and efficient by using binary
weights instead of real-valued weights. Current BNNs use latent real-valued
weights during training, where several training hyper-parameters are inherited
from real-valued networks. The interpretation of several of these
hyperparameters is based on the magnitude of the real-valued weights. For BNNs,
however, the magnitude of binary weights is not meaningful, and thus it is
unclear what these hyperparameters actually do. One example is weight-decay,
which aims to keep the magnitude of real-valued weights small. Other examples
are latent weight initialization, the learning rate, and learning rate decay,
which influence the magnitude of the real-valued weights. The magnitude is
interpretable for real-valued weights, but loses its meaning for binary
weights. In this paper we offer a new interpretation of these magnitude-based
hyperparameters based on higher-order gradient filtering during network
optimization. Our analysis makes it possible to understand how magnitude-based
hyperparameters influence the training of binary networks which allows for new
optimization filters specifically designed for binary neural networks that are
independent of their real-valued interpretation. Moreover, our improved
understanding reduces the number of hyperparameters, which in turn eases the
hyperparameter tuning effort which may lead to better hyperparameter values for
improved accuracy. Code is available at
this https URL

本文提出了一种基于高阶梯度过滤的新解释，用于解释二进制神经网络中的大小基于超参数的影响，并为其优化设计新的过滤器。通过这种改进的理解，我们可以减少超参数的数量和调整的努力，从而提高二进制神经网络的准确性。