Large Language Models (LLMs) have recently demonstrated a remarkable success
across various tasks. However, efficiently serving LLMs has been a challenge
due to its large memory bottleneck, specifically in small batch inference
settings (e.g. mobile devices). Weight-only quantization can be a promising
approach, but sub-4 bit quantization remains a challenge due to large-magnitude
activation outliers. To mitigate the undesirable outlier effect, we first
propose per-IC quantization, a simple yet effective method that creates
quantization groups within each input channel (IC) rather than the conventional
per-output channel (OC). Our method is motivated by the observation that
activation outliers affect the input dimension of the weight matrix, so
similarly grouping the weights in the IC direction can isolate outliers to be
within a group. We also find that activation outliers do not dictate
quantization difficulty, and inherent weight sensitivities also exist. With
per-IC quantization as a new outlier-friendly scheme, we then propose Adaptive
Dimensions (AdaDim), a versatile quantization framework that can adapt to
various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim
by augmenting prior methods such as Round-To-Nearest and GPTQ, showing
significant improvements across various language modeling benchmarks for both
base (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval)
LLMs.

大型语言模型（LLMs）在各种任务中取得了显著的成功，但在小批量推断设置下（例如移动设备），有效地为 LLMs 提供服务一直是一个挑战，因为其存在着大量的内存瓶颈。本文提出了一种量化方案，即基于权重的量化，但是 sub-4 bit 量化仍然是一个挑战，因为存在大幅度的激活异常值。为了减轻不良的异常值效应，我们首先提出了 per-IC 量化，这是一种简单而有效的方法，它在每个输入通道（IC）内创建量化组，而不是传统的每个输出通道（OC）。然后，我们提出了 AdaDim，这是一种可以适应各种权重敏感性模式的多功能量化框架。通过对先前的方法进行改进，如 Round-To-Nearest 和 GPTQ，我们展示了 AdaDim 的有效性，在基础的语言建模基准测试和指导性调优的 LLMs 中都取得了显著的改进效果（在 MMLU 上最高 + 4.7%，在 HumanEval 上最高 + 10%）。

重新思考通道维度，以隔离大型语言模型低比特权重量化中的异常值

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight  Quantization of Large Language Models

Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at this https URL soon.

我们提出了一种用于优化权重舍入任务的简洁高效方法，名为 SignRound，在 400 个步骤内实现了出色的结果，超过了四舍五入的基线方法，并与最近的方法有着令人印象深刻的竞争力，而不会引入额外的推断开销。