Quantization is a promising method that reduces memory usage and
computational intensity of Deep Neural Networks (DNNs), but it often leads to
significant output error that hinder model deployment. In this paper, we
propose Bias Compensation (BC) to minimize the output error, thus realizing
ultra-low-precision quantization without model fine-tuning. Instead of
optimizing the non-convex quantization process as in most previous methods, the
proposed BC bypasses the step to directly minimize the quantizing output error
by identifying a bias vector for compensation. We have established that the
minimization of output error through BC is a convex problem and provides an
efficient strategy to procure optimal solutions associated with minimal output
error,without the need for training or fine-tuning. We conduct extensive
experiments on Vision Transformer models and Large Language Models, and the
results show that our method notably reduces quantization output error, thereby
permitting ultra-low-precision post-training quantization and enhancing the
task performance of models. Especially, BC improves the accuracy of ViT-B with
4-bit PTQ4ViT by 36.89% on the ImageNet-1k task, and decreases the perplexity
of OPT-350M with 3-bit GPTQ by 5.97 on WikiText2.The code is in
this https URL

通过偏置补偿方法，本文提出了一种最小化输出误差的超低精度量化方法，无需模型微调，能够显著降低量化输出误差，并增强模型任务性能。

减小量化输出误差的偏差补偿

Minimize Quantization Output Error with Bias Compensation

Many applications of mobile deep learning, especially real-time computer
vision workloads, are constrained by computation power. This is particularly
true for workloads running on older consumer phones, where a typical device
might be powered by a single- or dual-core ARMv7 CPU. We provide an open-source
implementation and a comprehensive analysis of (to our knowledge) the state of
the art ultra-low-precision (<4 bit precision) implementation of the core
primitives required for modern deep learning workloads on ARMv7 devices, and
demonstrate speedups of 4x-20x over our additional state-of-the-art float32 and
int8 baselines.

通过对 ARMv7 设备上现代深度学习工作负载所需的核心基本操作进行开源实现和全面分析，我们展示了与现有市场上的 float32 和 int8 基准相比，使用大于 4 位精度的最先进的超低精度技术可获得 4 倍至 20 倍的加速度。