Post-Training Quantization (PTQ) enhances the efficiency of Large Language
Models (LLMs) by enabling faster operation and compatibility with more
accessible hardware through reduced memory usage, at the cost of small
performance drops. We explore the role of calibration sets in PTQ, specifically
their effect on hidden activations in various notable open-source LLMs.
Calibration sets are crucial for evaluating activation magnitudes and
identifying outliers, which can distort the quantization range and negatively
impact performance. Our analysis reveals a marked contrast in quantization
effectiveness across models. The older OPT model, which much of the
quantization literature is based on, shows significant performance
deterioration and high susceptibility to outliers with varying calibration
sets. In contrast, newer models like Llama-2 7B, Llama-3 8B, Command-R 35B, and
Mistral 7B demonstrate strong robustness, with Mistral 7B showing near-immunity
to outliers and stable activations. These findings suggest a shift in PTQ
strategies might be needed. As advancements in pre-training methods reduce the
relevance of outliers, there is an emerging need to reassess the fundamentals
of current quantization literature. The emphasis should pivot towards
optimizing inference speed, rather than primarily focusing on outlier
preservation, to align with the evolving characteristics of state-of-the-art
LLMs.

通过减少内存使用和提高操作速度，后训练量化（PTQ）能够增强大型语言模型（LLMs）的效率和与更多硬件的兼容性，尽管会导致一定的性能下降。然而，我们的研究发现在不同已知的开源 LLMs 中，校准集对于评估激活幅度和检测异常值至关重要，异常值可能扭曲量化范围并对性能产生负面影响。因此，我们建议重新评估当前量化文献的基础知识，从主要关注异常值保留转向优化推断速度，以适应现代化 LLMs 的特性。