The popularity of Convolutional Neural Network (CNN) models and the ubiquity
of CPUs imply that better performance of CNN model inference on CPUs can
deliver significant gain to a large number of users. To improve the performance
of CNN inference on CPUs, current approaches like MXNet and Intel OpenVINO
usually treat the model as a graph and use the high-performance libraries such
as Intel MKL-DNN to implement the operations of the graph. While achieving
reasonable performance on individual operations from the off-the-shelf
libraries, this solution makes it inflexible to conduct optimizations at the
graph level, as the local operation-level optimizations are predefined.
Therefore, it is restrictive and misses the opportunity to optimize the
end-to-end inference pipeline as a whole. This paper presents \emph{NeoCPU}, a
comprehensive approach of CNN model inference on CPUs that employs a full-stack
and systematic scheme of optimizations. \emph{NeoCPU} optimizes the operations
as templates without relying on third-parties libraries, which enables further
improvement of the performance via operation- and graph-level joint
optimization. Experiments show that \emph{NeoCPU} achieves up to 3.45$\times$
lower latency for CNN model inference than the current state-of-the-art
implementations on various kinds of popular CPUs.

本研究提出了一种名为 'NeoCPU' 的综合方法，通过对模板进行优化实现卷积神经网络模型的 CPU 推理，而不是依赖第三方库来实现模型图的单独操作优化，从而通过操作级别和图级别同时优化来进一步提高性能。实验结果表明，与当前各种流行 CPU 上的最先进实现相比，NeoCPU 的卷积神经网络模型推理延迟可降低 3.45 倍。