For fast and energy-efficient deployment of trained deep neural networks on
resource-constrained embedded hardware, each learned weight parameter should
ideally be represented and stored using a single bit. Error-rates usually
increase when this requirement is imposed. Here, we report large improvements
in error rates on multiple datasets, for deep convolutional neural networks
deployed with 1-bit-per-weight. Using wide residual networks as our main
baseline, our approach simplifies existing methods that binarize weights by
applying the sign function in training; we apply scaling factors for each layer
with constant unlearned values equal to the layer-specific standard deviations
used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with
1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve
error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We
also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test
results of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error
rates halve previously reported values, and are within about 1% of our
error-rates for the same network with full-precision weights. For networks that
overfit, we also show significant improvements in error rate by not learning
batch normalization scale and offset parameters. This applies to both full
precision and 1-bit-per-weight networks. Using a warm-restart learning-rate
schedule, we found that training for 1-bit-per-weight is just as fast as
full-precision networks, with better accuracy than standard schedules, and
achieved about 98%-99% of peak performance in just 62 training epochs for
CIFAR-10/100. For full training code and trained models in MATLAB, Keras and
PyTorch see this https URL .

该研究报告介绍了一种能够在资源受限制的嵌入式硬件上快速高效地部署训练完毕的深度神经网络的方法，即使用单个比特来表示和存储每个学习到的权重参数，通过在训练中为每个层应用等于层特定标准差的恒定未学习值的缩放因子来简化现有方法并提高误差率，并通过不学习批归一化缩放和偏移参数来改善网络性能。