Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

本文介绍了一种组合使用低精度计算和蒸馏知识来提高深度学习网络性能的方法，该方法取得了 ImageNet 数据集上 ResNet 架构各种变体的三元精度和 4 位精度的准确率的最新成果，并提供了三种应用蒸馏知识技术到训练和部署流程中的方案。

通过知识蒸馏技术提高低精度网络的准确度