Recent advancements in quantization and mixed-precision techniques offer
significant promise for improving the run-time and energy efficiency of neural
networks. In this work, we further showed that neural networks, wherein
individual parameters or activations can take on different precisions ranging
between 1 and 4 bits, can achieve accuracies comparable to or exceeding the
full-precision counterparts. However, the deployment of such networks poses
numerous challenges, stemming from the necessity to manage and control the
compute/communication/storage requirements associated with these extremely
fine-grained mixed precisions for each piece of data. There is a lack of
existing efficient hardware and system-level support tailored to these unique
and challenging requirements. Our research introduces the first novel holistic
hardware-software co-design approach for these networks, which enables a
continuous feedback loop between hardware design, training, and inference to
facilitate systematic design exploration. As a proof-of-concept, we illustrate
this co-design approach by designing new, configurable CPU SIMD architectures
tailored for these networks, tightly integrating the architecture with new
system-aware training and inference techniques. We perform systematic design
space exploration using this framework to analyze various tradeoffs. The design
for mixed-precision networks that achieves optimized tradeoffs corresponds to
an architecture that supports 1, 2, and 4-bit fixed-point operations with four
configurable precision patterns, when coupled with system-aware training and
inference optimization -- networks trained for this design achieve accuracies
that closely match full-precision accuracies, while compressing and improving
run-time efficiency of the neural networks drastically by 10-20x, compared to
full-precision networks.

最近，量化和混合精度技术的最新进展为改善神经网络的运行时间和能量效率提供了极大的希望。通过本研究，我们进一步证明了神经网络中的个别参数或激活可以采用 1 到 4 位不同精度，从而可以获得与全精度对应物相当甚至超过的准确性。然而，这类网络的部署面临着许多挑战，源于需要管理和控制与每个数据片的这些非常精细的混合精度相关的计算、通信和存储需求。现有的有效的硬件和系统级支持对于这些独特且具有挑战性的需求缺乏。我们的研究引入了首个针对这些网络的新颖的硬件 - 软件协同设计方法，该方法使得硬件设计、训练和推理之间具备连续的反馈循环，以促进系统设计探索。我们通过设计新的可配置的面向这些网络的 CPU SIMD 架构，并将该架构与新的系统感知训练和推理技术紧密结合，来说明这种协同设计方法的概念验证。我们使用该框架进行系统化的设计空间探索，以分析各种权衡。在该设计中，混合精度网络的优化权衡对应于一种支持 1、2 和 4 位定点运算的架构，该架构具有四种可配置的精度模式，当与系统感知训练和推理优化相结合时，该网络的准确性与全精度准确性非常接近，同时将神经网络的压缩和运行时效率大大提升 10-20 倍，相比全精度网络。