We present a novel optimization strategy for training neural networks which
we call "BitNet". The parameters of neural networks are usually unconstrained
and have a dynamic range dispersed over all real values. Our key idea is to
limit the expressive power of the network by dynamically controlling the range
and set of values that the parameters can take. We formulate this idea using a
novel end-to-end approach that circumvents the discrete parameter space by
optimizing a relaxed continuous and differentiable upper bound of the typical
classification loss function. The approach can be interpreted as a
regularization inspired by the Minimum Description Length (MDL) principle. For
each layer of the network, our approach optimizes real-valued translation and
scaling factors and arbitrary precision integer-valued parameters (weights). We
empirically compare BitNet to an equivalent unregularized model on the MNIST
and CIFAR-10 datasets. We show that BitNet converges faster to a superior
quality solution. Additionally, the resulting model has significant savings in
memory due to the use of integer-valued parameters.

BitNet 是一种基于 Minimum Description Length (MDL) 原则，通过动态限制网络参数的可取范围和值，优化训练神经网络的方法， 在 MNIST 和 CIFAR-10 数据集上通过优化实数值翻译和缩放因子以及任意精度整型参数，BitNet 比等效的非正则化模型更快地收敛生成了质量更高且内存消耗更小的模型。