We address the challenges associated with deploying neural networks on CPUs,
with a particular focus on minimizing inference time while maintaining
accuracy. Our novel approach is to use the dataflow (i.e., computation order)
of a neural network to explore data reuse opportunities using heuristic-guided
analysis and a code generation framework, which enables exploration of various
Single Instruction, Multiple Data (SIMD) implementations to achieve optimized
neural network execution. Our results demonstrate that the dataflow that keeps
outputs in SIMD registers while also maximizing both input and weight reuse
consistently yields the best performance for a wide variety of inference
workloads, achieving up to 3x speedup for 8-bit neural networks, and up to 4.8x
speedup for binary neural networks, respectively, over the optimized
implementations of neural networks today.

通过使用数据流和启发式引导分析及代码生成框架，我们致力于解决在 CPU 上部署神经网络时所面临的挑战，特别注重在保持准确性的同时最小化推断时间。我们的研究结果表明，通过保持输出在 SIMD 寄存器中的数据流，并同时最大限度地利用输入和权重重用，可以实现各种推断工作负载的最佳性能，使 8 位神经网络的速度提高 3 倍，二进制神经网络的速度提高 4.8 倍。