A low precision deep neural network training technique for producing sparse,
ternary neural networks is presented. The technique incorporates hard- ware
implementation costs during training to achieve significant model compression
for inference. Training involves three stages: network training using L2
regularization and a quantization threshold regularizer, quantization pruning,
and finally retraining. Resulting networks achieve improved accuracy, reduced
memory footprint and reduced computational complexity compared with
conventional methods, on MNIST and CIFAR10 datasets. Our networks are up to 98%
sparse and 5 & 11 times smaller than equivalent binary and ternary models,
translating to significant resource and speed benefits for hardware
implementations.

本文提出了一种低精度深度神经网络训练技术，用于生成稀疏三元神经网络，该技术在训练期间考虑硬件实现成本，以实现重要的模型压缩，结果网络精度提高、内存占用和计算复杂性减少，与传统方法相比，在 MNIST 和 CIFAR10 数据集上，我们的网络稀疏度高达 98％，比等价二进制和三进制模型小 5＆11 倍，从而在硬件实现方面具有重大资源和速度优势。