Despite recent advances in LLM quantization, activation quantization remains
to be challenging due to the activation outliers. Conventional remedies, e.g.,
mixing precisions for different channels, introduce extra overhead and reduce
the speedup. In this work, we develop a simple yet effective strategy to
facilitate per-tensor activation quantization by preventing the generation of
problematic tokens. Precisely, we propose a method to find a set of key-value
cache, coined CushionCache, which mitigates outliers in subsequent tokens when
inserted as a prefix. CushionCache works in two steps: First, we greedily
search for a prompt token sequence that minimizes the maximum activation values
in subsequent tokens. Then, we further tune the token cache to regularize the
activations of subsequent tokens to be more quantization-friendly. The proposed
method successfully addresses activation outliers of LLMs, providing a
substantial performance boost for per-tensor activation quantization methods.
We thoroughly evaluate our method over a wide range of models and benchmarks
and find that it significantly surpasses the established baseline of per-tensor
W8A8 quantization and can be seamlessly integrated with the recent activation
quantization method.

提出了一种名为 CushionCache 的方法，通过防止生成问题标记，来促进每个张量的激活量化，成功解决了 LLMs 的激活离群值问题，并为每个张量的激活量化方法提供了显著的性能提升。

添加前缀注意力节点以缓解大型语言模型量化中的激活值异常

Prefixing Attention Sinks can Mitigate Activation Outliers for Large  Language Model Quantization

Large language models (LLMs) with hundreds of billions of parameters show
impressive results across various language tasks using simple prompt tuning and
few-shot examples, without the need for task-specific fine-tuning. However,
their enormous size requires multiple server-grade GPUs even for inference,
creating a significant cost barrier. To address this limitation, we introduce a
novel post-training quantization method for weights with minimal quality
degradation. While activation outliers are known to be problematic in
activation quantization, our theoretical analysis suggests that we can identify
factors contributing to weight quantization errors by considering activation
outliers. We propose an innovative PTQ scheme called outlier-aware weight
quantization (OWQ), which identifies vulnerable weights and allocates
high-precision to them. Our extensive experiments demonstrate that the 3.01-bit
models produced by OWQ exhibit comparable quality to the 4-bit models generated
by OPTQ.

本文提出了一种后训练量化方法，可以在不损失质量的情况下，在模型中针对 weight 使用较高的精度，大大降低了模型推理需要的 GPU 数量，实现了更高的经济性。