Post-training quantization (PTQ) has emerged as a promising technique to
reduce the cost of large language models (LLMs). Specifically, PTQ can
effectively mitigate memory consumption and reduce computational overhead in
LLMs. To meet the requirements of both high efficiency and performance across
diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to
guide the selection of quantization methods. This paper presents a thorough
evaluation of these factors by evaluating the effect of PTQ on Weight,
Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon,
Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with
parameters ranging from 125M to 180B. The evaluation encompasses five types of
tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context
tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization
methods to demonstrate their applicability. Based on the extensive experiments,
we systematically summarize the effect of quantization, provide recommendations
to apply quantization techniques, and point out future directions.

本论文通过对 PTQ 技术在 11 个模型家族的综合评估，系统总结了量化对权重、激活函数和 KV Cache 的影响，提供了应用量化技术的建议，指出了未来的研究方向。