High throughput and low latency inference of deep neural networks are
critical for the deployment of deep learning applications. This paper presents
the efficient inference techniques of IntelCaffe, the first Intel optimized
deep learning framework that supports efficient 8-bit low precision inference
and model optimization techniques of convolutional neural networks on Intel
Xeon Scalable Processors. The 8-bit optimized model is automatically generated
with a calibration process from FP32 model without the need of fine-tuning or
retraining. We show that the inference throughput and latency with ResNet-50,
Inception-v3 and SSD are improved by 1.38X-2.9X and 1.35X-3X respectively with
neglectable accuracy loss from IntelCaffe FP32 baseline and by 56X-75X and
26X-37X from BVLC Caffe. All these techniques have been open-sourced on
IntelCaffe GitHub1, and the artifact is provided to reproduce the result on
Amazon AWS Cloud.

本文介绍了 IntelCaffe 的高效推断技术以及用于深度学习模型优化的技术，它是第一个支持 8 位低精度推断的 Intel 优化深度学习框架，能够在 Intel Xeon Scalable 处理器上加速卷积神经网络的模型优化过程。该模型的 8 位优化模型可通过一次标定过程从 FP32 模型自动生成，无需进行微调或重新训练，其在 ResNet-50、Inception-v3 和 SSD 上的推断吞吐量和延迟分别提高了 1.38X-2.9X 和 1.35X-3X，精度损失可忽略不计，与 IntelCaffe FP32 基线相比，这些技术还使得吞吐量和延迟分别提高了 56X-75X 和 26X-37X。这些技术已在 IntelCaffe GitHub 进行了开源，并提供了 Amazon AWS Cloud 上重现结果的文物。