Low-precision quantization is recognized for its efficacy in neural network
optimization. Our analysis reveals that non-quantized elementwise operations
which are prevalent in layers such as parameterized activation functions, batch
normalization, and quantization scaling dominate the inference cost of
low-precision models. These non-quantized elementwise operations are commonly
overlooked in SOTA efficiency metrics such as Arithmetic Computation Effort
(ACE). In this paper, we propose ACEv2 - an extended version of ACE which
offers a better alignment with the inference cost of quantized models and their
energy consumption on ML hardware. Moreover, we introduce PikeLPN, a model that
addresses these efficiency issues by applying quantization to both elementwise
operations and multiply-accumulate operations. In particular, we present a
novel quantization technique for batch normalization layers named QuantNorm
which allows for quantizing the batch normalization parameters without
compromising the model performance. Additionally, we propose applying Double
Quantization where the quantization scaling parameters are quantized.
Furthermore, we recognize and resolve the issue of distribution mismatch in
Separable Convolution layers by introducing Distribution-Heterogeneous
Quantization which enables quantizing them to low-precision. PikeLPN achieves
Pareto-optimality in efficiency-accuracy trade-off with up to 3X efficiency
improvement compared to SOTA low-precision models.

低精度量化在神经网络优化中得到广泛应用，非量化的逐元素操作主导了低精度模型的推理成本，本文提出了 ACEv2 以更好地衡量量化模型的推理成本和能耗，并介绍了一种名为 QuantNorm 的批归一化层的新型量化技术，以及应用双量化解决量化缩放参数的问题以及引入 Distribution-Heterogeneous Quantization 来解决可分离卷积层中的分布不匹配问题，PikeLPN 在效率和准确性的权衡中实现 Pareto 优化，并比 SOTA 低精度模型提高了 3 倍的效率。