Post-training quantization of Large Language Models (LLMs) is challenging. In
this work, we introduce Low-rank Quantization Error Reduction (LQER), which
combines quantization and low-rank approximation to recover the model
capability. LQER leverages an activation-induced scale matrix to drive the
singular value distribution of quantization error towards a desirable
distribution, which enables nearly-lossless W4A8 quantization on various LLMs
and downstream tasks without the need for knowledge distillation, grid search,
or gradient-base iterative optimization. Unlike existing methods, the
computation pattern of LQER eliminates the need for specialized Scatter and
Gather processes to collect high-precision weights from irregular memory
locations. Our W4A8 LLMs achieve near-lossless performance on six popular
downstream tasks, while using 1.36$\times$ fewer hardware resources than the
leading state-of-the-art method. We will open-source our framework once the
paper is accepted.

本研究提出了一种通过低秩近似和量化相结合的方法，使用激活引起的比例矩阵将量化误差的奇异值分布调整到理想值，实现了在各种大型语言模型和相关任务上几乎无失真的 W4A8 量化，而无需知识蒸馏、网格搜索或基于梯度的迭代优化，并且与现有方法相比，计算模式中的 LQER 消除了从不规则内存位置收集高精度权重所需的专用 Scatter 和 Gather 过程，从而在六个热门任务上实现了接近无失真性能，同时使用的硬件资源比领先的最先进方法少 1.36 倍。

LQER: 低秩量化误差重建法用于 LLMs

LQER: Low-Rank Quantization Error Reconstruction for LLMs

Internet of Things (IoT) sensors are nowadays heavily utilized in various
real-world applications ranging from wearables to smart buildings passing by
agrotechnology and health monitoring. With the huge amounts of data generated
by these tiny devices, Deep Learning (DL) models have been extensively used to
enhance them with intelligent processing. However, with the urge for smaller
and more accurate devices, DL models became too heavy to deploy. It is thus
necessary to incorporate the hardware's limited resources in the design
process. Therefore, inspired by the human brain known for its efficiency and
low power consumption, we propose a shallow bidirectional network based on
predictive coding theory and dynamic early exiting for halting further
computations when a performance threshold is surpassed. We achieve comparable
accuracy to VGG-16 in image classification on CIFAR-10 with fewer parameters
and less computational complexity.

借鉴人脑的高效率和低功耗，本文提出了一种基于预测编码理论和动态提前终止的浅双向网络，以加入硬件有限资源在设计过程中，实现在 CIFAR-10 图像分类上与 VGG-16 相当准确率的结果，使用更少的参数和较低的计算复杂度。