Compressing large language models (LLMs), often consisting of billions of
parameters, provides faster inference, smaller memory footprints, and enables
local deployment. Two standard compression techniques are pruning and
quantization, with the former eliminating redundant connections in model layers
and the latter representing model parameters with fewer bits. The key tradeoff
is between the degree of compression and the impact on the quality of the
compressed model. Existing research on LLM compression primarily focuses on
performance in terms of general metrics like perplexity or downstream task
accuracy. More fine-grained metrics, such as those measuring parametric
knowledge, remain significantly underexplored. To help bridge this gap, we
present a comprehensive analysis across multiple model families (ENCODER,
ENCODER-DECODER, and DECODER) using the LAMA and LM-HARNESS benchmarks in order
to systematically quantify the effect of commonly employed compression
techniques on model performance. A particular focus is on tradeoffs involving
parametric knowledge, with the goal of providing practitioners with practical
insights to help make informed decisions on compression. We release our
codebase1 to enable further research.

压缩大型语言模型（LLM）包含数十亿参数，可以提供更快的推理速度，更小的内存占用，并支持本地部署。我们通过对多个模型系列（ENCODER、ENCODER-DECODER 和 DECODER）使用 LAMA 和 LM-HARNESS 基准进行全面分析，以系统量化常用压缩技术对模型性能的影响，特别关注涉及参数化知识的权衡，旨在为从业人员提供实用的见解，帮助他们在压缩决策时做出明智的选择。