Large Language Models (LLMs) have recently demonstrated a remarkable success
across various tasks. However, efficiently serving LLMs has been a challenge
due to its large memory bottleneck, specifically in small batch inference
settings (e.g. mobile devices). Weight-only quantization can be a promising
approach, but sub-4 bit quantization remains a challenge due to large-magnitude
activation outliers. To mitigate the undesirable outlier effect, we first
propose per-IC quantization, a simple yet effective method that creates
quantization groups within each input channel (IC) rather than the conventional
per-output channel (OC). Our method is motivated by the observation that
activation outliers affect the input dimension of the weight matrix, so
similarly grouping the weights in the IC direction can isolate outliers to be
within a group. We also find that activation outliers do not dictate
quantization difficulty, and inherent weight sensitivities also exist. With
per-IC quantization as a new outlier-friendly scheme, we then propose Adaptive
Dimensions (AdaDim), a versatile quantization framework that can adapt to
various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim
by augmenting prior methods such as Round-To-Nearest and GPTQ, showing
significant improvements across various language modeling benchmarks for both
base (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval)
LLMs.

大型语言模型（LLMs）在各种任务中取得了显著的成功，但在小批量推断设置下（例如移动设备），有效地为 LLMs 提供服务一直是一个挑战，因为其存在着大量的内存瓶颈。本文提出了一种量化方案，即基于权重的量化，但是 sub-4 bit 量化仍然是一个挑战，因为存在大幅度的激活异常值。为了减轻不良的异常值效应，我们首先提出了 per-IC 量化，这是一种简单而有效的方法，它在每个输入通道（IC）内创建量化组，而不是传统的每个输出通道（OC）。然后，我们提出了 AdaDim，这是一种可以适应各种权重敏感性模式的多功能量化框架。通过对先前的方法进行改进，如 Round-To-Nearest 和 GPTQ，我们展示了 AdaDim 的有效性，在基础的语言建模基准测试和指导性调优的 LLMs 中都取得了显著的改进效果（在 MMLU 上最高 + 4.7%，在 HumanEval 上最高 + 10%）。